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FOREWORD 

When the U.S. Office of Education was chartered in 1867, one charge to 
its commissioners was to determine the nation's progress in education. The 
( National Assessment of Educational Progress (NAEP) was initiated a century 
later to address, in a systematic way, that charge. 

Since 1969, the Natioiial Assessment has gathered infomation about levels 
of educational achievanent across the country and reported its findings to the 
nation. It has surveyed the attainments of 9-year-olds, 13-year-olds, 
177year-olds and adults in art, career and occupational development, 
citizenship/social studies, mathematics, music, reading/literature, science 
and writing. All areas have been periodically reassessed in order to detect 
any important changes, lb date. National Assessment has interviewed -and tested 
more than 900, COO young Americans. 

Learning-area assessments evolve from' a consensus process. Each 
assessment is the product of several years of work by a great many educators, 
scholars and lay persons from all over the nation. Initially, these people 
design objectives for each subject area, proposing general goals they feel 
Americans should be achieving in the course of their education. After careful 
reviews, these objectives are given "to exercise (item) writers, whose task it 
is to create measurement instruments appropriate to the objectives.- 

When the exercises have ^ passed extensive reviews by subject-matter 
specialists, measurement experts and lay persons, they are administered to 
probability .samples. The people v*io compose these samples are chosen in such a 
' -fey that the results of their assessment can be generalized to an entire 
national population. That is, on the basis of the performance of about 2,500 
9-year-olds on a given exercise, we can make general izatiions about the 
probable performance of all 9-year-olds in the, nation. 

^ After assessment data ^ave been collected, scored and analyzed, the 
National Assessment publishes retxurts and disseminates the results as widely 
as possible,. Not all exercises are^'^leased for publication. Because NAEP 
will readminister some of the same exercises in the future to determine 
u*iether the .performance level of Anericans has increased, remained stable or 
decreased, it is essential they not be released in order to preserve the 

integrity of the stuSy- 
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INTRODUCTION 

I 

The National Assessnent of Eaixrational Progress (NAEP) has caapieted 
three assessments of writing, the first conducted in 1969-70, the second in 
1973-74 and the third during 1978-79. Each assessment surveyed the writing 
achievement of American 9-, 13- and 17-year-olds, using a deeply stratified, 
multistage probability sample design. This report documents procedures used 
in the 1978-79 writing assessment and also describes changes in procedures 
between assessnents. 

lb measure the status of writing achievement in 1978-79, * National 
Assessment consultants first reviewed and refined the objectives used in the 
second writing assessment. Consultants then worked with staff to develop 
exercises. Most items (exercises)-^ developed for the 1978-79 assessment will 
be kept secure and used for measuring changes in performance in future 
assessments, but some will be released following the 1978-79 data collection. 

Approximately 22,500 9-year-olds, 30,500 13-year-olds and 27,500. 
17-year-olds participated in the 1978-79 writiix| assessment. Because each 
item booklet is administered to a representative sample at an age group and 
because data are reported only for groups of students, not individuals, each 
student assessed did not respond to all items. Each student completed one 
item booklet of about 45 minutes in length. Between 2,100 and 2,700 students 
responded to each booklet. 

The exercises for each assessment were administered by a professional 
data collection staff to minimize the burden on participatiix| schools and to 
maximize uniformity of assessment conditions. Instructions and items were 
recorded on a paced audio tape and played back to students to reduce the 
potential effect of reading difficulties and to insure that all students moved 
through the booklets at the same speed. 

Since National Assessment staff and consultants feel strongly that 
writing performance should be assessed on the basis of writiix| samples rather 
than objective tests, most of the writing exercises are open-ended, requiring 
students to produce a piece of writing. Responses to open-ended items were 



National Assessment uses the term ''exercise" to mean an assessment item. The 
terms "exercise" and "item" are used interchangeably in this report. 

%or the 1978-79 assessment, 17-year-olds sampled included only those 
attending school at the time of the assessment. 
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hand-scored by trained readers using detailed scoring guides; the few 
multiple-choice items included were scored by an optical scanning machine. 
Several types of open-ended scoring were conducted — holistic^ primary traits 
essay cohesion, paragraph coherence, syntax and mechanics. For each type of 
scoring procedure, papers collected in different assessnents were landomly 
ordered into a single pool and scored simultaneously fcllowing iche 1978-79 
assessment. Different types of scoring were done by different groups of 
scorers. 

When a report indicates that "85% of the 17-year-olds gave a particular 
response," it means that an estimated 85% of the 17-year-olds would have given 
that response if ail the 17-year-olds in schools across the country had been 
assessed. National Assessment reports of writing results provide estimates of 
percentages of responses at each score level or some combination of score 
levels for each writing task. Increases or decreases in these percentages 
between assessments are used to indicate trends in achievement. Tb provide 
descriptive information about syntax and mechanics, data are presented that 
illustrate the average perfonnance as well as the range of performance. In 
addition to reporting national results, National Assessment provides data on 
the performance of various population subgroups within the national 
population, defined by sex, race, region of the country, size and type of 
community lived in, level of parental education, grade in school and other 
categories as appropriate. 

The following chapters describe specific procedures used to develop 
objectives and exercises, draw the assessment sample, prepare materials for 
the assessment, administer and score the items and analyze the results. 
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CHAPTER 1 



OBJECTIVES REDEVELOPMENT 

The primary goal of the National Assessment of Educational Progress 
(NAEP) is to repoLt on the current educational statue of young Americans and 
to monitor any changes in achievement over time* For each learning area to be 
assf:ss*.ed^ NAEP asks consultants to develop objectives that define the subject 
area. Since the objectives provide guidelines for exercise developers, 
consul tants a re asked to incl ude exampl es of the knowl edge , skills and 
attitudes to be assessed at each age level. 

Education in Anerica is a collaborative enterprise involving a great many 
people^ with vddely differing philosophies. Providing information about 
education nationwide wuld be considerably easier if there were consensus 
about the means and ends of Anerican education, but the fact is that Anericans 
have conflicting and sometimes contradictory values regarding the goals of 
education and the means for achievir^ thOT. To develop an- assessment that is 
truly national in scope and takes into account the diversity of fcurricula, 
values and goals across the country. National Assessment employs a consensus 
process for developing objectives, with representation of many different 
groups of people. . 

Several types of consultants help to develop National Assessment 
objectives. College and university specialists in a learning area insure that 
the objectives include important concepts that the schools should be teaching. 
Educators, including classroom teachers, curriculun supervisors and persons 
involved in teacher education, make sure that the objectives describe 
concepts, skills and attitudes that the school^ should be teaching and those 
that they presently are teaching. Concerned citizens, parents and other 
interested lay persons must agree that the objectives are important for young 
people to achieve, are free of educational jargon and are not biased or 
offensive to any groups. Consultants are selected to represent different 
regions of the country, minority groups, various types of communities and age 
levels. 

T^ble 1 presents the objectives used for the second and third writing 
assesanents. The objectives for the 1973-74 assessment* also served as the 
basis for the 1978-79 writing assessment, since restricted funds and limited 
development time made a full-scale revision effort impractical. However, the 
objectives were refined based on NAEP's experiences with the second writing 
assessment and comments from users of NAEP materials. 

\ 

Objectives II and III are based on the writing objectives developed for 

1 
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the 1969-70 assessment uj the Eaucational Testing Service (ETS) ^ Princeton^ 
New Jersey^ in 1965. The original writing objectives were reviewed by writing 
experts and lay panels and used as the basis for generating exercises for thr 
first assessment of writing. 



TABLE 1. Objectives for the 
1973-74 and 1978-79 Writing 
Assessments 

I. Demonstrates ability in writing to reveal personal feelings and 
ideas 

A. Through free expression 

B. Through the use of conventional modes of discourse 

II. Demonstrates ability to write in response to a wide - range of 
societal demands and obligations. Ability is defined to include 
correctness in usage^ punctuation^ spelling and form or convention 
as appropriate *".o particular writing tasks^ e.g.^ manuscripts, 
letters 



A. Social 

1. Personal 

2. Organizational 

3. Cbmmunity 

B . Busi ness/vocat ional 

C. Scholastic 

III. Indicates the importance attached to writing skills 

A. Recognizes the necessity of writing for a variety of nieeds (as 
in Objectives I and II) 

B. writes to fulfill those needs 

C. Gets sat isf action , even enjoyment, from having written 
something well 



Redevelopment of objectives for the second (1973-74) assessment of 
writing was also done by ETS. Reccxnmendations from consultants indicated that 
creative or personal writing as well as expository writing should be a part of 
the assessnent. Thus, Objective I was added. Also, lay reviewers felt that 
the mechanical aspects of writing, such -as grammar, punctuation and spelling, 
should be included in the revised description of Objective II, although they 
agreed that mechanical correctness was not to be regarded as the only 
criterion for judging a piece of writing. 
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A recurring concern has been the difficulty and appropriateness of 
measuring an objective that requests respondents to reveal personal feelings. 
One suggestion was that Objective I might apply to personal writing done with 
feeling; another was that the objective might simply require the expression of 
feelings (whether they were "fictional" or "real"). National Assessment has 
elected to interpret Objective I broadly as the ability to engage in writing 
for expressive purposes. 

Prior to the developnent of second assessment exercises, National 
Assesanent staff and consultants decided that writing, particularly public 
writing of the sort described in Objective II, is done for a particular 
audience and should be evaluated in view of its intended effect on that 
audience. This decision necessitated the development of narrowly defined 
tasks that required writing to specified audiences for particular purposes, 
iiie major purpose of most social, business and scholastic writinn was 
identified as either persuasive or explanatory. Thus, for the second and third 
assesanents of writing, • liJational Assessment developed Objective II writing 
tasks to measure either explanatory or persuasive writing appropriate to 
specific social, business or scholastic situations. 

As a result of current assessment procedures and available resources, the 
last twD assessments have concentrated on Objectives I and LI. This is not 
meant to imply that Objective III is unimportant. 

For a complete description of the writing objectives ax\] procedures used 
to develop fiem, see Writing Objectives, Second Assessment (1972). 



CHAPTER 2 
DEVELOPMENT OF EXERCISES 



The Exercise Development 
Framework 

National Assessment's major task is to describe changes over time in the 
educational attainments of students. lb do this effectively, NAEP must 
collect information about what things students know and can do in a variety of 
subjects. To provide such information about writing skills. National 
Assesanent found it necessary to develop a system for assessing writing that 
would provide data about specific skills within a reasonable budget. For the 
first assesanent of writing, essay tasks were scored holistically. This 
approach to scoring involves the use of several broad scoring categories with 
papers ranked according to the general quality of an essay taken as a whole. 
However, this approach did not provide sufficient descriptive information 
about specific writing skills for National Assessment's purposes. 

Prior feo the second assessment of writing, NAEP staff held a conference 
of writing %aucators and measurement specialists, ihese specialists inoicated 
that both evaluative and descriptive information about writing were needed. 
Since most writing is done with a purpose in mind, they proposed that the 
assesanent concentrate on measuring the writing abilities needed to perform 
particular tasks, ihus, beginning with the 1973-74 writing assessment, NAEP 
has used the Primary Trait System (PTS) for scoring essays, which evaluates 
the capacity to write for precisely defined purposes. In this system, an 
overall evaluative rating on a 4-point scale indicates level of achievement of 
the primary trait defined for the writing task — that is, how successful each 
paper is in reaching its purpose. Scoring for secondary characteristics, such 
as control of tense, coherence, mechanics or consistency of voice, provides 
descriptive data. 

• Such an evaluation system has implications for exercise and scoring guide 
development. First, to insure that the writing tasks require a range of 
skills, exercise developers, need guidelines about the kinds of tasks that 
should be developed. Accordingly, National Assessment staff held conferences 
with writing educators to refin^ the objectives and provide new guidel:.nes for 
writing task developnent. Objective I was interpreted as the ability to engage, 
m writing^ for expressive purposes, while Objective II was interpreted as 
requiring persuasive or explanatory writing. It was agreed that each age group 
should be asked to write at least one expressive, one persuasive and one 
explanatory task, and that the tasks would be varied over the years, l^sks 
developed to measure expressive writing have included narratives. 
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role- playing., fantasy, expression of emotion and humorous writing. Persuasive 
and ■ explanatory tasks have required letters, speeches and reports appropriate 
to various social, business and scholastic situations — for example, a "thank 
you" letter, an accident report, a speech to a community or a letter 
correcting a billing error* ; 

Second, each exercise must clearly explain the writing task required. It 
must be specific as to the role of the writer, the purpose of the 
communication and the intended aiidience, and the evaluative criteria to be 
used by scorers should be made apparent to the writer. For example, if papers 
are to be judged on expressive qualities, writers should be instructed to try 
to be expressive. Furthermore, questions and rhetorical situations should be 
phrased in such a way that alJ respondents will give comparable answers. 

Third, to insure that responses can be evaluated in a way that provides 
information about the skill the exercise is attempting to measure, the rating 
criteria must be developed along with the exercise* The primary trait score 
le'/els and, when possible, procedures for rating designated secondary 
characteristics, should be identified in advance of field testing and then 
refined using sample papers. 



1978-79 Exercise Developroervt 

Exercises were developed by persons active in the field of writing 
education. The main developnental effort focused on primary trait system 
exercises designed to assess the skills necessary for effective expressive, 
explanatory and persuasive writing. Since National Assessment does not have 
the resources to assess a wide variety of tasks at any one time, there vyas an 
effort to cover some areas hot emphasized in earlier assessments. There was 
also an effort to include some items' that examined use of commonly accepted 
strategies for improving writing. For instance, one item asked 13- and 
17-year-olds to revise their wrk after writing a draft; another asked them to 
"prewrite" or jot down ideas before beginning to write. 

In designing exercises for the 1978-79 writing assessment , developers 
also investigated" the possibility of using more multiple-choice items' tc 
assess writing skills. Some items assessing ability to locate the central 
idea of a paragraph and ability to identify -words needed to make up correct 
grammatical structures in sentences were developed but not assessed because 
consultants indicated that this type of information could be obtained from 
standardized test data. Consultants felt that assessment resources .would be 
better devoted to obtaining information about these skills from actual writing 
samples; thus, two new scoring systems, one for cohesion and one for 'syntax, 
were developed (see Chapter 6). ' 

TWO types, of items other than essay tasks developed for the 1978-79 
assessment did prove successful: sentence combining tasks and paragraphs with 
conn€?ctive ties clozed. These exercises were designed not to measure specific 
objectives but rather to provide additional information about particular 
writing skills and to help in ^^loviding a context for reporting changes in 
writing performance, 

6 • ^ 
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The sentence combining tasks required respondents to combine two- or more 
simple sentences into a longer sentence that meant the same thing. In the 
cloze paragraph items, a paragraph vas coiiStructed to contain a nunBer of 
pronoun references and specific transitional devices. Several of these 
references and connective ties were clozed (deleted and replaced by blanks) , 
and respondents were offered a nunber of multiple-choice options to replace 
each blank. 

Finally, to find out about students' attitudes toward writing and to 
•provide some instructional context for reporting results, an attitudinal scale 
and additional background questions were developed. 



Released Writing Tasks 

About half of the 1978-79 writing assessment at each age consisted of 
reassessed items already given in previous assessments so that measuranents of 
changes in performance could be made. Several of these items appeared in all 
three assessments; others were administered only in the second and third 
assessments- 



The last of the items developed for the first assessment and most of the 
^ Items a^essed in both 1973-74 and 1978-79 have been released and published in 
Report lO-W-25, The Third Assessment of Writing: 1978-79 Released Exercise Set 
(1981). Some items developed for the 1973-74 assessment and most of the new 
Items developed for the 1978-7y assessment will be kept secure and used to 
measure changes in achievement in future assessments. T^ble 2 shows essay and 
letter writing items released after the third assessment and the yea.s in 
which they were assessed. Each task is 'labeled with a shor; name and unique 
item identification nunber. ID numbers beginning with "1" indicate the iten: 
measures expressiv- ■ writing; nunbers beginning with "2" indicate items 
measuring explanatory or persuasive writing. 

Sentence combining and cloze paragraph items were assessed for the first 
time in 1978-79; about half of these have been released and the remainder will 
be used to measure change. 



Item Development, Field Test, 
Review and Selection 
'Procedures 

, Itetr; writers wrked in small groups to create items, and the groups then 
critiqued each others' work. Once items had been developed, critiqued and 
revised, they were reviewed by National Assessment staff. The results of 
these reviews were compiled and item writers once again revised items. 
Survivi.ng items were field tested in schools across the country to discover 
potential problems in wording, directions or administrative procedures and to 
collect item statistics, timing information and scoring information. Schools 
were selected to represent high-- and low-income communities as well as more 
typical communities. The tryouts .v.-ere administered to students in at least 
four classrooms (approximately 100 students) at each of the ages assessed. 




TABLE 2* Essay and Letter Wtiting Exercises 
Released FDllowing 1978-79 Assessment 

Assessed in 1969-70, 1973-74 and 1978-79 

Age 9 Fireflies (102012) 

Kangaroo (102013) 
Age 13 Rainy Day (102015) 

Age 17 Stork (102016) 

Ages 13 and 17 Ciescribe Something (203012) 

Assessed in 1973-74 and 1978-79 

Age 9 Goldfish (101006) 

Pappy Letter (201001) 

Age 13 Loss (101007) 

^ Principal Letter (201006) 

Age 17 Grape Peeler (101015) 

Rec Center (201007) 
Electric Blanket (202014) 

Assessed in 1978-79 

Ages 9 and 13 Poster Calendar (202031) 



So that the^ield tests would closely simulate actual assesanent field 
procedures, the students recorded their ansv^srs in the test booklets; 
directions and questions were read to students froin an audio tape arid National 
Assessment staff members, rather than classroom teachers, administered the 
test. Ihe students* responses to the items, as well as the administrators" 
reports of any field problems, helped Both staff and consultants to evaluate 
and revise the exercises. Revised exercises were generally field tested 
again. 

After exercises vsre field tested, the results were reviewed by NAEP 
staff and panels of writing educator*: and lay persons from across the country* 
Exercises for each age group were reviewed for appropriateness by teachers v*io 
taught students at that age.-^ Lay citizens, representing a -variety of 
occupations and interests, also reviewed the exercises, checking for sex or 
racial bias and considering the geoeral importance of each exercise. A panel 
of writing specialists worked with staff to examine the items judged to be 
successful by the review panels and then- made the final selection of the items 
included in the 1978-79 assessment. ^ 
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CHAPTER 3 



PREPARATION OF ASSESSMENT MATERIALS 



Preparation of Booklets 
and Audio Tapes 

National Assessment uses a matrix sampling approach, with different 
nationally representative samples of students respondir^ to different item 
booklets (see Chapter 4 for details). Since the Assessment's aim is to 
describe results for groups of students (males, blacks, students in the Vfest, 
and so on) , not individuals, it is not necessary for each student to respond 
to all the items. Each student responded to one booklet of items designed to 
be completed in a single class period. 

The 1978-79 assessment vas a combined assessment of writing, music and 
art. Because of the length of many of the writing items, booklets of exercises 
included either writing and music exercises or writing and art exercises. 
Following the selection of exercises to be included in the assessment. 
National Assessment staff determined which exercises were to be included in 
the various booklets and sequenced them within the booklets. Booklets were 
constructed separately for each age level since students at different ages 
received different sets of exercises. Ihus, exercises for 9-year-olds were not 
sequenced in the same order as those for 13^ear-olds, and so forth. In 
1978-79, there were- 9 exercise booklets that contained writing exercises for 
9-year-olds, 11 such booklets at age 13 and 10 such booklets for 17^ear-olds. 

The following constraints were observed in preparir^ the 1978-79 exercise 
booklets: 

1. Each booklet contained exercises of varying difficulty so that 
students uould not become bored by many easy exercises or discouraged 
by many difficult exercises. 
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Exercises could not cue other exercises. In other words, the answer 
to one exercise could not be contained in another exercise in the 
same booklet. 

Each booklet was timed so that it would take no more than 45 minutes 
— * the length of a typical class period — of a student's time. 
Booklets cont:a4ned approximately 30-35 minutes of exercise time and 
an additional 18-15 minutes of introductory material, instructions 
and background questions. 
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4, Booklets were designed to be, insofar as possible, parallel with 
respect to the nonber of different objectives measured and difficulty 
levels. Items measuring a particular objective were scattered 
throughout the booklets so that many different students would respond 
to questions related to a particular objective. 

National Assessment has constantly attempted to institute procedures to 
minimize difficulties connected vith the testing siLuation so that results 
will be, as nearly as possible, an accurate reflection of what students know 
and can do. Considerable effort was devoted to developing c^ear instructions 
for procedures students should vse in answering items and to formating the 
booklets to help students writo as well as possible in the assessnent 
situation. For example, space "was provided for students to write their answers 
directly in the assesanent booklets, not on separate answer sheets. It was 
felt that this procedure would reduce possibilities for confusion in using 
additional sheets of paper, especially for the younger students. Also, to 
reduce the possibility of fatigue effects, essay items were placed in the 
beginning of booklets so students could write earlier rather than later in the 
administrative session, lb minimize guessing, students were encouraged to 
select the "I don't know" response option included with multiple-choice items. 

Paced audio tapes were prepared for each exercise booklet. Most of the 
written portions of an exercipe stimulus and response options were read aloud 
to reduce the effect of any reading difficulties and to insure that all 
students moved through the booklets at the same speed. In addition, the use of 
tapes helped to insure uniform assessment conditions across the country. 

In 1973-74 and 1978-79, special instructions were given to item 
administrators regarding the longer (15-25 minutes) writing tasks. If the 
students appeared to be finished with an item well before the allotted time 
had been used, administrators were asked to announce that students had several 
minutes to finish up and then to move on to the next item. As ic turned out, 
administrators seldom used this procedure but felt it u^s extremely important 
to have such an option for those cases Oien a group did finish early. National 
Assessment has found that if the administrator cannot proceed, the respondent:; 
may get restless or even leave the assessment session. 



Differences in Item Booklets 
±n the Three Writing 
Assessments 

National Assessment makes every effort to make assessment conditions for 
items measuring change identical from assessment to assessment so that any 
changes observed will be attributable to changes in achievement rather than a 
response to an altered testing condition. Although items were kept identical, 
the makeup of the item booklets was different for each writing assessment. In 
1969-70, writing items were assessed with science and citizenship itans; in 
1973-74, writing. and career and occupational developnent items were included 
in the same booklets; in- 1978-79, writing items shared booklets with either 
art or music iteit^s. Typically only one writing essay was included in an item 
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booklet. Although some essays were given at the end of administrative sessions 
in the first assessment^ writing essays were consistently placed very close to 
the beginning of the booklets in the second and third assessments, ihi- change 
was made to reduce the effects of differences in context from assessment to 
assessment. 



CHAPTER 4 
SAMPLING • 



This chapter describes procedures used in designing and selecting the 
sanples for the three writing assessments. Sample design and selection for the 
wr icing assessmeats were conducted by the Research Triangle Institute , 
Raleigh, North Carolina, and monitored by NAEP staff. 

Ihe tarqet populations for each of the assessments included 9-, 13- and 
17-year-olds^ enrolled in either public or private schools at the time of the 
assessment vjio were not functionally handicapped to the extent that they could 
not participate in ah assessment, specific groups excluded were: non-Engltsh- 
speaking persons, those, identified as nonreaders, persons physically or 
mentally unable to respond, and persons in institutions or attending schools 
established- for the physically or mentally handicapped.^ 

National Assessment did not follow up specific individuals from one 
assessment to the next. In other words, the students v*io participated in the 
firi^t writing assessment were not the same ones v*io participated in the second 
or third assessment. However, in each assessment year, participants were 
carefully selected to represent each age level. Fbr example, at? age 9 
different sets of probability samples were used for the three assessments, but 
each set .contained, nationally representative samples of the population of 
students v*io were nine years old during that assessment year. Thus, if we say 
that 9-year-olds' achievement improved between 1974 and 1979, we mean that 
students v*io were nine years old in 1979 correctly answered the same questions 
more often than those who were nine years old in 1974. 

Ihe NAEP samples were designed to provide approximately 2,000 to 2,700 
respondents per writing task. These nunbers allowed reporting of data for the 
nation and for the subgroups defined in Appendix A. Ihe definitions of the 
target populations were identical in each assessment; however, the design used 
to obtain representative samples of these populations was modified following 
the second writing assesanent. 

Several essays were scored for mechanical errors and use of various 
syntactic devices in addition to primary trait or cohesive scoring. While NAEP 



Definition of 1978-79 assessment .age groups are: 9-year-olds ~ born during 
calendar year 1969; 13-year-olds — born during calendar year 1965; -and 
17-year-olds — born October 1, 1961, throiqh September 30, 1962. 
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feels that descriptions of mechanics and syntax provide 'val-uable information, 
such detailed scoring is also quite expensive. Accordingly, only subsamples of 
responses to several essays were scored in this fashion. Use of a subsample 
.allows reporting of results for the nation but for only a limited nunber of 
subgroups. , ^ 

An overview of the National Assessment approach to sample design is given 
.below, followed by a description of f .ocedures specific to the three writing 
assessments and a description of the -procedures used to obtain. the subsamples 
of papers. . ; . - . ■ ^ 

Overview of the National 
• , Assessment Sample Design. 

For all of its assessments. National Assessment uses a deeply stratified, 
three-stage national probability sample design with oversampling of low-income 
and rural areas. In the first stage, the United States^ is divided into 
geographical units of counties'- or groups of coritiguous, counties -meeting a 
minimun population size ^requirement. These units, called primary sampling 
units (PSUs) , are stratified by region ^^nd size of community. From the list of 
PSUs, a sample of PSUs is drawn without -replacement with probability 
proportional to population size measures, representing all regions and sizes 
of communities. Oversampling of low-income and extrane-rurai areas is first 
. performed at this stage by adjusting the estimated pppulation-size measures of 
such areas to increase sampling rates. In the current sampling procedures. 
Census Einployment Survey Data-- are used within PSUs to further delineate and 
oversample low-income areas. Counties with high proportions of rural families 
•ar-e also oversampled. . 

In the second stage, all public and : private schools within each ^'PSU 

selectt?d in the first stage are iistedt' Schools within each PSU are selected 

.without replacement with probabilities proportional to the ^nunber of 
age-el igibles injc±ie school. 

The third stage of sampling occurs during the data collection period. A 
list of all age-eligible students within « each selected school is mad-e. A 
simple random selection of eligible students without replacement is- obtained 
and item booklets are administered to selected students. Specially trained 
personnel select the sample and administer the booklet3. " - 

Each respondent In the sample does' not^ have the same probability of 
selection because some subpopulations are * oversajripled and because adjustments 
are made to compensate for student nonresponse arid for some schools' refusal 
to participate. The selection probability for^ each ioHividual is computed,- 
and its^ reciprocal is used to weight 'each response in any statistical 
calculation to compensate for unequal rates of sampling and to insure proper' 
representation in the population structure. 

The nunber of PSUs, schools within PSUs and students within schools^are' 
determined by optimun sampling principles. That is, a sample design is 
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utilized that will achieve the maximun precision for a given level of 
resources* 

Table 3 displays ,the nimber of PSUs used and the nunber of schools in 
which assessment sessions were conducted by age for each of the three wri'iing 
assessments. 



TABLE 3. Number of PSUs Selected and Schools 
Within PSJs Included in the Assessment 
in 1969-70, 1973-74 and 1978-79 





1969- 


-70 


1973- 


-74 


1978- 


-79 




Assessment 


Assessment 


Assessment 




No. of 


No. of 


No. of 


No. of 


No. of 


No. of 




PSUs 


Schools 


PSUs 


Schools 


PSUs 


Schools 


Age 9 


204 


935 


116 


1,246 


75 


539 


Age 13 


205 


749 


116 


1,278 


75 


496 


Age 17 , 


193 


670 


116 


1,052 


75 


435 



Changes in Sample Design : 
1969-70, 1973-74 and 1978-79 

A major change in sampling procedures has been a reduction in the target 
popilations with" each subsequent assessment. In 1969-70 and 1973-74, 
17-year-olds who were not currently attending school were included in the 
assessment. 'These 17-year-olds were sampled by getting their names from school 
li^ts of dropouts and early graduates. Young , adults aged 26-35, identified 
through a household sample, were assessed in the 1969-70 assessment. 
Assessment of out-of-school 17-year-olds and adults is quite expensive, and in 
1978-79 funds to assess these groups were not available. 

Procedures' for selecting the samples of students attending school have 
changed somewhat between the writing assessments. For the 1978-79 sample, two 
types of PSUs were identified:, (1) large-size population areas defined by the 
U.S. Bureau of the Census as Standard Metropolitan Statistical Areas (SMSAs) 
and (2) other* contiguous non-SMSA counties grouped together to meet certain, 
minimum size requirements. Ihe first stratification of PSUS was by geographic 
region, as defined by the Office of . Business Economics, U.S. Departinent of 
Commerce. (See ^pendix A for a definition of regional subgroups.) 

Within regions, ^PSUs were classified into five si ze-of -community (SOC). 
categories: . * 

SOC 1 PSUs corresponding to the 13 largest SMSAs after adjusting the 
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population size to compensate for oversampling low-income 
metropolitan areas. These PSUs have selection probabilities so 
large that under NAEP allocation procedures they are certain 
to be included in the sample each year. These PSUs are 
designated as self-representing . 

PSUs corresponding to the remaining 57 SMSAs with over SOOyOOO 
population. 

PSUs corresponding to the remaining 162 SMSAs. 

PSUs made up of non-SMSA counties. SOCs 4 and 5 are determined 
so that half of the remaining population (after adjustment for 
oversampling of rural areas) falls into each category. SOC 4 
contains PSUs in which less than 60% of the residents are 
classified as rural. 

The self-representing PSUs represented additional stratification, making 
an effective total of 17 (13 + 4) size-of-community strata. Each 
self-repiresenting a»1SA was divided further into geographical substrata or 
nonoverl.apping replicates that cons';ituted multiples of convenient work units 
for item administration. These multiple work units were included with the rest 
of the nonself-representing PSUs to form the. pool from which first-stage 
sampling units were selected. Tb insure adequate representation. National 
Assesanent doubled the sampling rate of low-income and rural areas.- 

In 1975-76, first-stage units were selected simultaneously for four 
consecutive a'ssessment years (1975-76 through 1978-79), as were schools in the 
self- representing PSUs. The sample design required that every four years NAEP 
assess at' least once in every state and not more than once in any school. 
There -were 1/101 primary sampling units in the primary sampling frame for the 
fQur-year period, from which about 75 first-stage sampling units were selected 
each year. - 

Each year, within the primary strata public and private* schools were 
Listed and further stratified by the estimated nunber of youngsters in a 
school eligible at each age. Shiall schools were clustered until they formed a 
large enough group to respond to the same number of item booklets as the 
larger schools in a stratun. Schools or school clusters were selected without 
replacement, with probability proportional to the nunber of age-eligibles in. 
the school or cluster of schools. Chce^ schools were identified, districts 
were contacted to check for changes in grade range and for the existence of 
new .schools. This information was used to revise probabilities ;of schools' 
selection.. 

In the third stage, students within each sample school were selected with 
equal probability and without replacement. The nunber of students selected was 
proportional to the -punber .of age-eligibles, with oversampling in -JLow-income 
and rural areas. 'In .1978-79, the size of groups sel*ected for administrative 
sessions within schools was allowed to vary from 10 to 25 students. This 
enabled National Assessment to obtain desired sample sizes in schools having 



SOC 2 

SOC 3 
SOCs 4&5 
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characteristically low response rates. This feature also permitted 
last-minute modifications and adjustments to selection probabilities 
necessi*:ated by enrollment changes. 

The sampling procedures used in the 1969-70 and 1973-74 assessments 
differed somev+iat f rom<^ ^ose used in^ the 1978-79 assessment ( Final 
Report. . .Sampling and Weighting Activities. .. ^ 1980; Moore et al., 1974; 
Chromy and Horvitz, 1970). First, size measures for SMSAs, counties and urban 
areas in 1978-79 were based on 1970 census data, while those in 1969-70 were 
based on 1960 census data. Size measures in 1973-74 were based on 1960 census 
data and first-count data from the 1970 census. 

Another difference occurred in the PSU sample design. In 1969-70, PSUs 
were stratified by region, size of community, a measure of socioeconomic 
status (SES) and geographic proximity. There was no requirement that all 
states be included in the sample. In 1973-74, the PSU' were stratified by 
region and size of ccxmnunity. In addition, the sample ;*Jas constrained to 
include all states. In 1978-79, PSUs were stratified by region and size of 
community, with the constraint that each state must appear in the sample once 
very four years. 

SOC categories were defined as shown previously in both the second and 
third assessments, but the number of SMSAs included in SOCs 1, 2 and 3 changed 
between 1973-74 and 1978-79. In% 1973-74 , there were 15 self- representing SMSA 
PSUs in SOC 1, 50 SMSAs in SOC 2 and 178 SMSAs *in SOC 3. 

The size-of-community stratifications used .in 1973-74 and ^1978-79 
differed from those used in the 1969-70 assessment. Jn 1969-70, there were 
only four SOC stratifications. The first SOC category consisted of all central 
cities with overall population greater than 180,000; the second included the 
remainder of the SMSA containing the central city in^SOC 1. SOC 3 consisted of 
the other SMSAs and' ail coun-ties not included in SOC 1 or 2 that contained at 
least one city with a population '^over 15,000. SOC 4 consisted' of all counties 
not included in SOC 1, 2 or 3. - . 



^ In 1973-74 and 1978-79, oversampling of low-income metropolitan areas and 
extreme-rural areas was accomplished at the primary stage by increasing the 
estimated population size measures of PSUs containing these areas and then 
sampling with probabilities 'propor*- ' nal to these adjusted size measures. In 
1969-70, a. poverty index was use o stratify PSUs into high- and low-SES 
stratifications. The sampling rates within these strata were then increased in 
order to achieve the desired oversampling. In the third stage, 12 students 
were selected for each administrative session. . . 



Sampling of Essay Responses 
for Mechanics Scoring 

The two essays at each age that had been given in all three writing 
assessments were designated for detailed mechanics and syntax scoring. 



17 



CYie of these essays at each age had been analyzed and released for 
publication following the 1973-74 assessment. Fbr thic essay, the scoring 
process involved replicating procedures used in 1973-74 to enable reporting of 
data from three assessments. Approximately 400 responses at an age were 
subsampled from the total nunber of papers collected in each assessment, 
allowing reporting of results for the nation and for males and females. 
Selection of sample responses from the first two assessments was done in 
1973-74 using random sampl ing. techniques. For those assessments, booklets 
containing the items were administered to groups of about 12 students, in 
sampling, booklets were ordered by student ID nunber and two booklets were 
chosen^ from each group administration using a table of random nunbers. 
^Following the third assessment, a systematic sample of one-fifth of the 
1978-79 papers was drawn at each age. Papers were ordered by ID nunbers within 
schools and every fifth response was chosen. For 9-year-olds, 5>eiection began 
with the fourth paper in the ordered list; for 13-year-olds, with ^Ke fifth 
paper; and for 17-year-olds, with the second paper. 

Ihe other essay at each age scored for mechanics and syntax was not 
ana.'yzed^ and released until the third assessment. For these essays, an 
opoortunity existed to revise the sample size, and it was decided to report 
data on blacks' and whites' performances as well as that of the nation and of 
males and females. Accordingly, a larger sample — approximately f^50 — was 
drawn for each age with oversampling of blacks. A separate systenatic sample 
was drawn for blacks and nonblacks. At ages 9 and 13, one of every two blacks 
and one of every five nonblacks was chosen. At age 17, due to a smaller nunber 
of responses per booklet, 5 of each 9 blacks and 3 of each 13 nonblacks were 
selected, l^ble 4 shows the sampling patterns used for these essays. 



TABLE 4. Systematic Sampling Patterns for Mechanics 
and Syntax Scoring of Essays With Oversampling of Blacks 

Age 9 Age 13 Age 17 

Blacks 1 of 2 1 of 2 5 of 9 

Select 2nd of each pair 1st of each pair 1, 3, 5, 7, 9 

of each 9 

Nonblacks 1 of 5 1 of 5 3 of 1^ 

Select .4th of each 5 2nd of each 5 3, 7, 11 of 

of each 13 



The nunbers of students responding to each booklet containing writing 
exercises are shown in Chapter 5, and the sample sizes used in obtaining 
writing assessment results for the various scoring procedures for essay items 
of each assessment are detailed in l^ble 5. 
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TABLE 5i Sample Sizes Used for Reporting 
Writing Assessment Results 

Age Exercise Analysis No* in Sample 

1969-70 1973*74 1978-79 





"Kangaroo" 


Holistic scorin>; 










A (\A 




(narration) 


Paragraph coherence 




384 




409 


494 






Syntax and mechanics 




384 




409 


494 


13 


"Describe" 


Holistic scoring 




395 




420 


536 




(descrip- 


Pa rag rap*! coherence 




395 




420 


536 




tion) 


Syntax and mechanics 




395 




420 


536 


1 / 


Describe 


Holistic scoring 




Job 




Hi. 1 


b Jo 




(descrip- 


Paragraf*! coherence 




365 




417 


538 




tion) 


Syntax and mechanics 




365 




417 


538 




Fire tlies 


Primary trait scoring 2, 


4bb 


2, 


b 1 i 


*i ceo 

2, bbi 




(narration) 


Cohesion 




466 


2, 


573 


2,553 






Syntax and mechanics 




556 




584 


596 


13 


"Rainy Day" 


Primary trait scoring 


2. 


408 


2, 


621 


2,804 




(expression) 


Cohesion 


2. 


408 


2, 


621 


2,804 






Syntax and mechanics 




589 




630 


680 


17 


"Stork" 


Primary trait scoring 


2. 


073 


2, 


281 


2,748 




(narration) 


Cohesion 


2. 


073 


2, 


281 


. 2,748 






Syntax and mechanics 




594 




596 


722 


9 


•^3o Id fish" 


Primary tr . : t scoring 






2, 


611 


2,475 




(expression) 














9 


"Puppy 


Primary trait scoring 






2, 


643 


2,494 



Letter" 
(persuasion) 

9 "Poster Primary trait scoring 2,492 

Calendar" 
(explanation- 
business letter) 

13 "Loss" Primary trait scoring 2,607 2,775 

(expression) 

13 "Principal Primary trait scoring 2,552 2,793 

Letter" 
(persuasion) 
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TABLE 5 (Continued). Sa-nple Sizes Used 
for Reporting Writing Assessment Results 

13 "Poster Primary trait scoring 2,776 

Calendar" 
(explanation- 
busi ness 
letter) 

17 "^rape Primary trait scoring 2,283 2,765 

Peeler" 
(expression- 
humorous) 

17 "Recreation Primary trait scoring 2,308 2,784 

Crnter" 
(persuasion) 

17 "Electric Primary trait scoring 2,776 2.781 

Blanket" 
(explanat ion- 
bus i ness 
letter) 

13 Background 29 430 

questions ' 

17 Background 34,211 26,651 

questions 



The attitudinal questions, as well as some sentence-combining and cloze 
exercises were administered to all three age groups. All were administered 
for the first time in 1978-79, and all were administered and analyzed for full 
samples of 2,400-2,700 respondents at each age. 
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DATA COLLECTION 



National Assessment subcontracted data collection to the Research 
Triangle Institute and the Westinghouse DataScore Systems, Iowa City, Iowa, 
for the 1969-70 writing assessment and to the Research Triangle Institute for 
the 1973-74 and 1978-79 assessments. A professional data collection staff was 
used rather than school personnel to minimize the burden on participating 
schools and to insure, insofar as possible, uniform administration conditions 
across the country (Final Reports, >In-Schao3 Field Operations- , 1979). In 
all three assessments, NAEP staff worked closely with the subcontractors to 
insure adherence to rigorous administrative standards. 

Participation in the National Assessment is voluntary. NAEP makes every 
effort to encourage the schools selected in the sample to participate in the 
assessment, and National Assessment and Research Triangle Institute staffs 
have obtained high rates of school cooperation, as showi in T^ble 6 (Final 
Report. . . In-School Field Operations. , . , 1979, p. 39, T^ble 27). Student 
cooperation rates were also high. Percentages of the student sample covered 
are discussed in Appendix C. T&ble 7 shows the actual nimber of students v*io 
responded to the different exercise booklets at each age level in the 1978-79 
assessment. 



TABLE 6. School Cooperation Rates, 
1978-79 Assessment 



Age Percent of Eligible Schools 

Participating in 1978-79 
Assessment 

9 . 90.4 

13 90.9 

17 92.9 

Overall 91.3 



^Formerly Measurement Research Center,- lova City, lova, 
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1MLE ?• Number of Students Responding to Each 
It an Booklet in 1978-79 Assessment, by Age 





9 




Age 13 




Age 17 


Booklet NUirber 


Booklet Number 


Booklet Number 




Responding 










1 


2,532 


1 


2,755 


1 


2,730 


2 


2,^^53 


2 


2,801 


2 


2,746 


3 


2,475 


3 


2,775 


3 


2,761 


4 


2,494 


4 


2,791 


4 


2,772 


5 


2,479 


5 


2,785 


5 


2,684 


6 


2,522 


6 


2,748 


6 


2,739 


7 


2,531 


7 


2,736 


7 


2,642 


8 


2,524 


8 


2,779 


8 


2,656 


9 


2,486 


9 


2,754 


9 


2,787^ 


10 


2,483;!; 


10 


2, 758 


10 


2,697"^ 


11 


2, 526"^ 


11 


2,751 


11 


2,628 






12 


2,720^^ 


12 


2,628"^ 






13 


2,757"^ 


13 


2,698"^ 










14 


2,654"^ 


Total 


27,605 




35,910 




37,822 



"'"ihere were no writing exercises included in booklets 10 and 11 at age 9, 
booklets 12 and 13 at age 13 and booklets 10, 12, 13 and 14 at age 17. 



Each age group vas assessed at approximately the same time of the school 
year in each of the three writing assessments: 13-year-olds were assessed in 
October-December, 9-year-olds in January-February and 17-year-olds in 
March-May. In 1978-79, booklets were administered to groups of 10-25. 
students, with each group responding to only one of the booklets for their age 
level. Ihe groups varied in size depending on an^ estimate of the rate of 
nonresponse for a particular school. In 1969-70 and 1973-74, the planned 
session sizes were fixed at 12 students. 

In each assesanent, steps were taken to guarantee the anonymity of 
respondents. Students' names were listed with their booklet identification 
number so that scoring and processing personnel could go back to the school 
lists for data verification — for instance, on background information — if 
necessary, ihese lists did not leave the schools and were destroyed six months 
following the assessment in a school. 

To provide information on respondents' backgrounds, school officials were 
asked to respond to a "principal's questionnaire," which included questions 
about the size and type of community served by the schools. In additix^n, in 
1978-79, officials in 13- and 17-year-olds' schools were asked to respond to a 
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"supplementary principal's questionnaire/' which asked about writing programs 
in the school. Students also provided information on their backgrounds 
through questions included in the item booklets. Samples of forms used to 
collect background infonnation from students and school officials in the 
1978-79 assessment appear in Appendix B. Changes from the two earlier 
assessments are described on the forms. 

The assessment administrator coded each student's birth date, sex, grade, 
racial/ethnic classification and identification nunber on his. or her^ booklet. 
Administrators made a visual racial/ethnic identification at the time each 
student turned in his or her booklet. During the 1978-79 assessment, six 
different racial classifications were used: white, black, Spanish heritage, 
American Indian or Alaskan native. Pacific Islander or Asian, and 
unclassi f ied . If an administrator was unsure of a student' s racial/ethnic 
group, the administrator referred to the student's name or listened to the 
student talk to make the identification. The assessment administrators did not 
ask students to give a racial identification for themselves; however, in 1979, 
17-year-old students were asked to provide this information in one of the 
background questions included in the exercise booklet. 

" Sample sizes of the classifications Afnerican Indian or Alaskan native and 
r -jcific Islander or Asian were too small to permit reporting for these groups. 
Also, results for the group classified as Spanish heritage cannot be reported 
for separate exercises, only cor aggregates of exercises. Since the number of 
writing exercises was small, the primary analysis procedure for writing was to 
report results for individual exercises; thus, data .for Hispanic students were 
not provided.- . 

Following data collection, assessment administrators sent completed 
booklets to the scoring contractor, Westinghouse DataScore Systems, Iowa City, 
Iowa. Booklets were quality checked to verify that correct administrative 
procedures were followed by the field staff. Cbded identification information 
was also checked for accuracy; inconsistencies that could not be^ reconciled 
were sent back to the assessment administrator to be checked against the list 
of student names and identification nunlDers retained by the school for six 
months following the assessment. 
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CHAPTER 6 



SCORING 



Trained scorers classified responses to essay and other open-ended items 
using detailed scoring guides^ while responses to the few multiple-choice 
items in the writing assessment were read directly by optical-scanning 
machines. Scoring and computer recording - of data were contracted to 
Westinghouse DataScore Systems, Iowa City, Iowa, ^f or all three of the' writing 
assessments . 

National Assessment has found it most efficient to have scoring done by 
an outside contractor and to have the same contractor do both the machine 
scoring and the open-ended, or hand, scoring. Booklets to be scored do not 
have to be shipped to another location when different scoring methods are 
needed; in addition, the scoring contractor has a trained staff of scoring 
personnel chat can be called upon and augmented when National Assessment 
conducts a major scoring effort. 

Several types of scoring were ,used for the 1978-79 assessment. The 
procedures used wi'th each exercise are listed in Chapter 4, T&ble 5. Each is 
briefly described below. Complete docun?ntation of all exercises released 
^after the ; third assessment of writing including scoring guides and sample 
responses is contained in Report lO-W-25, The Third Assessment of Writing; 
1978-79 Released Exercise Set (1981). Readers desiring detailed information 
about the scoring procedures should consult the released exercise set, as well 
as Mullis (1980), Nkallis and .>fellon (1980) and Brown (1979). 

Overview of Methods Used in 
Scoring Writing 

Primary Trait System 

Almost all of the essays and letters included in the writing assessment 
were evaluated using the primary trait system of scoring. This system 
describes a respondent's ability to choose and effectively carry out 
appropriate rhetorical strategies. 

National Assessment uses a 4-point scale to describe levels of 
proficiency in the primary skill being assessed. Generally, level "1" 
indicates little or no evidence of the skill; level "2" indicates minimal 
evidence of the skill; level "3" indicates solid performance or competence; 
and level "4" is reserved for very good performance. Each essay task has a 
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scoring guide specifically tailored to the primary trait being measured. For 
example, explanatory writing should present facts in a clear and orderly 
fashion. At the lower end of the scoring categories would be a response with 
no information, clarity or logical order, while at the upper end would be an 
informative and unified presentation. 

In addition tr- primary traits, NAEP fathered information about sone 
secondary traits, si»ch as tone or letter format. Such traits were generally 
categorized as present or not present ~ for example, a letter either did or 
did not contain a greeting, closing, and so forth. 

lb insure accuracy of change measures, responses collected in different 
assesanents were scored at the same time. Because scores given in. an earlier 
assessment, even if the same scoring guides were used, might reflect slightly 
different judgment criteria, responses to unreleased writing tasks generally 
were not scored immediately following an assessment, but were held until the 
item was designated to be released- Then, responses from all assessments in 
which the item v^as included were scored together. 



Holistic Scoring 

Holistic scoring was used to rate one exercise at each age that had been 
included in all three writing assessments. When readers holistically score 
papers, they do not focus upon particular aspects of a paper such as 
mechanics, ideas or organization. Rather, they concentrate upon forming an 
overall impression of each paper relative to the other papers they have read. 
Since their primary task is to rank the papers from best to worst, this type 
of scoring tends to encourage a normal distribution. Thus, it is essential 
that sets of papers be scored together if results are to be compared because 
the distribution of holistic scores for any given subset of papers is highly 
dependent upon the entire set of papers with which it is scored. If two sets 
of papers are scored separat'-ly, the distribution of scores for each set may 
be ciiite different from what it WDuId be if the two sets were scored together. 

For each age. National Assessment randomly ordered papers collected in 
all three^ years into a single pool and conducted a single scoring session. 
This scoring^ vs^s subcontracted to Edward White, University of California at 
San Bernardino. Readers had general guidelines describing four levels of 
quality in which papers could be categorized. 

Since readers did not know which papers had been collected in which 
assessnent, they necessarily applied the same criteria to all the papers. 
After the scoring session, the ratings were examined to determine v^ether 
groups of papers written in different years had received better or worse 
rankings than each other. 

For holistic scoring, papers were read by only one scorer. Reliability 
of the scoring was checked by having a random 10% of the papers read by two 
scorers. The scorers agreed on 68% of the papers for age 9 and 79% of the 
papers for both ages 13 and 17. 
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Since the essays scored using the holistic method had been analyzed and 
released following the 1973-74 assessnient, responses from the first two 
assessments were scored together at that time. Following the 1978-79 
assessment, responses from the two previous assessments were rescored with the 
1978-79 responses. It should be noted that scores previously reported for the 
first and second assessment papers will differ from the scores reported for 
those papers following the third assesanent because they were reevaluated 
within a different total set* of papers. 



Coherence r Syntax and Mechanics 

In addition to judging the overall quality of responses. National 
Assessment exam i ned spec i f ic el ements o f coherence , mechanics and syntax ; 
however, such scoring can be quite time-consuming and exp^ensive. Thus, even 
though National Assessment recognizes the importance of specific and concrete 
information about these skills, it generally applies these types of scoring to 
only one or tw writing tasks at an age level and leaves many of the analyses- 
needed 'to supply detailed descriptive and diagnostic information to 
educational specialists and researchers. 

TTie systems for evaluating coherence, syntax and mechanics have evolved 
considerably since the first writing assessment. In the 1969-70 assessment, 
NAEP scorers counted mechanical errors and characterized papers on the basis 
of correct and faulty constructions. For the 1973-74 assessment. National 
Assessment wished to describe level of complexity and sophistication of 
writing as well as tabulate mechanical errors. Thus, categorization of 
paragraph coherence and sentence types was added. 

TTie paragraj* coherence, mechanics and syntax scoring systems developed 
for the 1973-74 assessment were used with the subsamples of responses to the 
holistically scored exercises. Although these exercises were scored and 
released for publication following the second assessment, they were reassessed 
in 1978-79 to provide additional information about changes in performance 
across three points in time. A comparative subsample of responses from the 
third assessment was scored using the 1973-74 guidelines for paragraph 
coherence, mechanics and -sentence types. (Details on the way in which the 
subsamples of responses were selected appear . in adapter 4 and sample sizes 
appear in T^ble 5.) Since these guidelines were more objective than those for 
the holistic scoring^ it was decided that all first and second assessment 
responses did not have to be rescored with third assessment data, lb insure 
that scoring was consistent, a reliability study \yas done, in which 10% of the 
responses from both the first and second assessment's were rescored with third 
assessment responses. Agreement across the various categories averaged about 
90%. However, for a few categories, standard errors used to calculate changes 
in performance over time were increased by an adjustment factor based on the 
amount of disagreement to insure that changes in performance would not be 
attributable to changes in scoring across time. 

Following the 1973-74 assessment, NAEP responded to requests for more 
complete data on writing by conducting a T-^it analysis to provide more 
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accurate measures of syntactic development. Also, a new procedure, baspd on 
cohesive ties, was developed to describe cohetence. These scoring methods, in 
addition to the primary trait system and mechanics scoring, were used to rate 
responses for one previously unreleased exercise at each age that had been 
administered in all three assessments. For each of these scoring procedures, 
papers from all three assessments were randomly ordered and scored together. 

The term cohesion refers in general to the many ways words and ideas are 
linked together in writing to create a sense of wholeness and coherence. The 
cohesion scoring required readers to sort papers into groups representing four 
degrees of cohesiveness. Papers in the lowest group (level 1) display no or 
few connections between- sentences and are loosely structured. Papers in the 
next group (level 2) display attempts to tie- ideas ■ together here or there but 
do not show any unifying structure. Cohesii/e papers (level 3) display 
gathering and ordering of details and ideas, and fully coherent papers (level 
4) display a nintjer of strategies and devices that bind the paper into a 
unified whole. 

The T-unit analysis of syntax and the mechanics analysis were conducted 
only on a subsanple of responses to^these exercises.- (See Chapter 4 for a 
description of the way in which the subsamples were selected and T^ble 5 for 
sample sizes.) Syntax refers to the ways in which words are put together to 
form phrases, clauses and sentences. The T-unit analysis involved breaking 
each paper up into its "T-units" (a T-unit is a main clause with all its 
attendant modifying wards, phrases and dependent clauses) and examinirn the 
ways in which writers embedded information in T-units and joined- T-anits 
together. The mechanics analysis involved classifying aiid tabulating the kinds 
of errors writers made in sentence use, punctuation, spelling, and so forth. 

Sentence combining exercises, assessed only in 1978-79, were hand scored 
using scoring guides that provided a description of the various combining 
strategies that could be used. The attitudinal questions, cloze paragraph 
exercises and most of the background questions were multiple-choice and "were 
tabulated by machine. 

Training of Scorers 
and Scorer Reliability 

Westinghouse DataScore Systems and National Assessment staff worked 
together to train readers for all but the holistic evaluations discussed 
earlier. Different groups of scorers were used for the primary trait and 
cohesion scoring; however, training procedures were similar. In training 
sessions, readers were given the scoring guide for an item and responses that 
exemplified each scoring category. The reasons why responses were classified 
in particular categories were discussed; scorers' questions were answered and, 
if necessary, modifications were made to scoring guides. Readers th-^ 
several papers and categorizations were discussed. This proc ss 
until readers were familiar with the application of the scoring gui- 
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Using these training procedures anci careful monitoring during scoring'. 
National Assessnent has achieved excellent scorer agreement. In scoring actual 
assessment data, most papers were scored by two readers for primary trait and 
cohesion, f^r.y discrepancies in categorization we-'e resolved by a tfhLrd reader. 
The need for reconciliation occurred on less then 10% of the papers. Table 8 
gives the interscorer agreement percentages for primary trait and cohesion 
scoring on exercises that required more than one scorer, "percentages are for 



TABLE 8. Interscore*- Percentages of ^^reement 
for Primary Trait and Cohesion Scoring 
Conducted in 1978-79 



Age 9 

Fireflies (102012) 
Goldfish (101006) 
Puppy Letter (201001) 

Age 13 

Rainy Day (102015) 
Loss (101007) 
Principal 

Letter (201006) 

Age 17 

Stork (102016) 

Grape Peeler (101015) 

Rec Center (2C1007) 



Percents of Scorer Agreer^ent 
Primary Trait 



1969-70 
Papers 



93.3 



93.1 



96.7 



1973-74. 

Hr-iero 



94.3 
93.9 
93.6 



94. 
91. 



94.1 



93. 
96. 



1978-79 
Papers 



95.1 
94.7 

■93.3- 



95.2 



92.4 
91.8 

93.5 



95.4 
94.2 
91.2 



Percents of Scorer Agreement 
Cohesion 

1969-70 1973-74 1978-79 

Papers Papers Papers 

Age 9 

Fireflies (102012) 93.0 93.2 94.0 

Age 13 

Rainy Day (102015) ■• 91.5 93.5 95^9 
Age, 17 

Stork (102016) 93.1 94,0 • 94.2 
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the scoriiig conducted following the 1978-79 assessment of the randomly ordered 
pools of pdpers collected in the three d liferent assessments. They are broken 
out by assessment year to show the uniformity of agreement. 

Scoring procedures for »twD essay items (••Electric Blanket" and ••Poster 
Calendar") were so straightforvyard that only one reader was used. Sentence 
combining exercises also required scoring by only one reader. In these cases, 
to check reliability a random 10% of the papers were read by twD scorers. The 
percentages of agreement for the pairs of readers were about 99% for each 
task.. 

The mechanics and syntax, analyses were conducted at the same time by a 
group of readers thoroughly trained in grammar, usage and linguistics. 
Training procedures were similar to those used for the other scoring methods, 
but lasted about four ^ weeks due to the nunber and complexity of the 
categorizations. For mechanics and syntax categorizations, the problem was 
not so much with scorer disagreement on classifications as with ensuring that 
each paper had been fully read. Tb maintain reliable data when describing 
numerous and complex categories, papers were usually scored by at least three 
and sometimes by four or even five readers. 
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CHAPTER* 7"^ 
DATA ANALYSIS 



Measures of Achievement v 

The basic measure of achievement reported by National Assessment is the 
percentage of respondents answering a* given item accetptaBly* This percentage 
is an estimate of the percentage of 9-, 13- or l^*Vear-olds .v*io would respond 
acceptably to a given item if every 9-, 13- or 17-year-old iri the country were 
assessed. . . . * 

Percentages of acceptable responses are used because each item is 
designed as a separate mS^sure.of some aspect of an objective or subobjective. 
The purpose of National Assessment is to discover if more or fewer people are 
able to answer .these items acceptably — and thus meet the objectives — over 
time. 

Because of the nature of the writing assessment, exact definitions' of 
••correcf* or -"acceptable" performance are sometimes debatable. Thus, for the 
primary trait system (PTS) and cohesion evaluations, percentages were reported 
for eacjh response category. In addition, to help d,etennine v*iether or not 
performance had changed over^ time, percentages of better papers from each 
assessment were totaled and compared. Primary trait percentages were computed 
for both levels'" 2, 3 and 4 (marginal or. better) combined and levels 3 and 4 
- (good and very good) comttined. Cohesion percentages were 'only computed for 3 
and 4 combined, since level. 2/ papers were not considered really* cohesive. 
Holistic scoring- information vqs described in terms of percentages of papers 
at;, each score point" and average holistic score for each assessment year. 

Syntax and mechanics', scoring data were analyzed to provide results about 
changes in the range of* performance as well as the average .performance. A 
^ frequency distribution va's obtained for each characteristic (misspelled words 
per essay, .awkward sentences per essay,, subordination per T-unit, .and so 
forth) for e'^ch r>et of papers at an age for each assessment year. Means, 
medians, quarti^es and deciles 'were reported.. The means provided information 
about changes over niime in the average frequency with wiiich characteristics 
occurred. Medians, quartiles and deciles described the changes in the 
distributions of* each characteristic, giving the values below v*iich 10%, 25%, 
50%, 75% and 90% of the-pspers fell. . 

In addition to providing'- national results. National Assessment reported 
on the achievement of various subpopulations of interest. Groups are defined^ 
by region of the country/ sex, race, size and type of community lived in. 
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level of parents' education and grade in school (see Appendix A for 
definitions of these groups) . Results were al3o analyzed by frequency of 
writing done "in school and instructional experiences with writing. (See 
Appendix B for questions used ,to obtain information about instructional 
experiences with writing.) Average syntax and mechanics characteristics were 
computed for good and poor papers (as defined by holistic, primary trait and 
cohesion ratings) . 

Procedures for esti^nating percentages responding in various ways and 
averages of.^ different characteristics of the papers are dependent on the 
sample design. Each response by an ind.ividual was weighted and multiplied by 
an adjustment factor for nonresponse. An estimate of the percentages of a 
particular age .group that wouJd hive responded to a particular exercise in a 
particular way if the entire z^^e group were assessed was defined as the 
weighted nimber of that type of response divided by the weighted number of all 
the responses. ^A similar ratio of weights was used to estimate percentages and 
averages for reporting groups or subpopulations of interest. 

^ Ihe difference between percentages'* or averages for a reporting group and 
that of the entire age group (nation) on an exercise was used to describe the 
performance of any reporting group relative to the entire age group. This 
difference is a positive nunber if the group achieved a higher percentage or 
average than the entire age group and is a negative number if the group 
achieved a lower percentage or average. For example, a^group perfonnan<::e of 
+1.8% indicates that the percentage of respi&nses for the group is 1.8 
percentage points higher than the national perceiitage of responses for that 
age level. ; \ 

Increases or decreases in the percentages and averages betweeen two 
assessments were estimated by finding 'the diff^ence between percentages or 
averages obtained from each assessment. A positive difference indicates an 
increase, and a negative difference indicates a decrease in those v*io 
responded in a particular v^y from one assessment, (to the next. These 
differences, or change measures, were used to indicate i.rends in achievement 
for an age. level or subpopulation of interest. Changes in group differences 
from the nation ''yetwsen tvo assessments were used to indicate the relative 
trend of a group compared with the national trend of the age group. 

In considering National Assessment's achievement measures, it is the 
differences in performance between assessment years, among groups and among 
ages that are most useful.- By main€aining the same itan or set of items in 
making these comparisons, we' have a reasonable indicator of whether more or 



^pendix C discusses nonresponse in assessment samples. 

Following the 1978-79 assessment, a weighting7class adjustment procedure was 
used to smooth estimated population proportions across -the 10 assessments 
conducted between . 1969-70 and 1978-79. A discussion of this procedure is 
included in Appendix E. 
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fewer people know or can do sornething judged important. 



' ' Estimating Variability in 

Achievement Measures 

National Assessment used a national probability sample at each age level 
to estimate the proportion of people v*io would complete an exercise in a 
particular way. Ihe sample selected was one of a large nunber of all possible 
samples of the same size that could have been selected using the same sample 
design. Since an achievement measure computed from each of the possible 
samples' WDuld differ from one sample to another, the standard error of this 
statistic was used as a measure of the sampling variability among achievement 
measures from all possible samples. A standard error, based on one particular 
sample, served to estimate that sampling variability. 

In the interest of sampling and cost efficiencies, National Assessment 
used a complex, stratified, multistage probability sample design. Typically, 
complex designs do not provide for unbiased or simple computation of sampling 
errors. A reasonably good approximation of standard error estimates of 
acceptable response percentages and averages was obtained by applying the 
jackknife procedure (Miller, 1964, pp. 1594-1705; Miller, 1968,- pp. 567-82; 
Mosteller and TUkey, 1968) to first-stage sampling units within strata. 
iStandard errors for achievement measures such as national percentages, group 
differences, means or mean differences for a particular assessraent year were 
estimated .directly, taking advantage of featuc^s of the jackknife procedure 
that are generic to all . of these statistics. Since samples for different 
assessments are independent, the standard errors of the differences in 
achievement measures between assessments "can be estimated simply by the square 
root of the sum of squared standard errors from each of the assessments. 

The standard error provides an estimate of sampling reliability for the 
achievement measures used by National Assessment. It is comprised of sampling 
error and other random error associated with the assessment of a specific item 
or set of items. Random error includas all possible nonsystematic error 
associated^ with administer ir»g specific exercises to specific students in 
specific situations. Random differences among scorers for open-ended items are 
also included in the standard errors. 

National Assessment has adhered to., a standard convention whefeby 
differences between statistics are designated as statistically significant at 
the .05 level of significance. That is, differences in performance between 
assesanent yeers or between a reporting group and the nation are highlighted 
with asterisk^ only if they are at least twice as large as their standard 
error. « Differences'^this -large vould occur by chance in fewer than 5% of all 
possible repl-ications of our san»pling and data collection procedures for any 
particular'^reporting group or national estimates. 



See ^^pendix D for a more detailed description of National Assessment's 
computation of standard errors. 
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Controlling Nonrandom Errors 



Systematic errors can be introduced at any stage of an assessment — 
exercise development^ preparation of 'exercise booklets^ design or 
administrative procedur^s^ field administration^ scoring or analysis. These 
nonsampling^ nonrandom errors rarely can be quantified^ nor can the magnitude 
of the bias they introduce into the estimates be evaluated directly. 

Systematic errors can be controlled in large part by 'employing uniform 
administrative and scoring procedures and requiring rigorous quality control 
in all phases of an assessment. If the systematic errors are the same frcxn 
age to age or group to group^ then the differences in percentages or mean 
percentages are measures with reduced bias because subtraction tends to cancel 
the effect of the systematic errors. 

Similarly^ the effect of systematic errors in different assessment years 
can be controlled by carefully replicating in the second assessment the 
procedures carried out in the first. Differences in achievement across 
assessment years will also be measures with r^?duced bias since subtraction 
will again tend to cancel systematic errors. . ' ^ 

Although it is not possible, for every condition or procedure to remain 
exactly the same between assessments conducted several years apart/ National 
Assessment has made every effort to keep conditions as nearly the same as 
possible. Changes in procedures described in this report were judged to have a 
relatively minor impact. 
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CHAPTER .8 

REPORTS ABOUT THE WRITING ASSESSMENTS 

Each assessment generates a tremendous amount of infonnation. lb make 
the data as useful as possible to a variety of audiences, National Assessment 
provides several types of reports. 



Reports 

Since it is difficult and time-consuming to synthesize many discrete bits 
of data. National Assessment prepared reports about the writing assessments 
for the general public — including • parents, classroom teachers, school 
, administrators and legislators — that- not only provided considerable data 
about each exercise but also synthesized and highlighted assessment results. 

Although National Assessment does not interpret assessment results, it 
recognizes that data presented alone are oTten 'difficult to consider in 
perspective. Accordingly, National Assessment asked a group of writing 
educators to review and comment- upon t'le results; their comments are included 
in the reports summarizing the writing assessment results. 

Reports concerning changes in writing performance across the writing 
assesanents are organized by age level. The following reports are available: 

Writing Achie vement, 1969-79-; Results From the Third National Writing 
Assessment, Volune I — 17-Year-Olds , Report no. lO-W-Ol {1^80). 

Writing Achievement, 1969-79: Results From the Third National Wr iting 
Assesa ngHt, Volune II — 13-Year-Olds , Report no. 10-W-Q2 (19B0) . — : — 

Writing A chievement, 1969-79: Results From the Third National Writi ng 
Assessme nt , Volune III — 9-Year-Olds , Report no. lO-W-03 (19Rn) . 

An additional •repo'-c on the syntax used by. 9-, 13- and 17-year-old 
students is planned for publication during 1981. 

The following reports describing results from the second assessment' are 
also available: , 

Writin9 Mechanics; 1969-1974; a Capsule Descripti on of Changes in 
Writing Mechanics , Report no. 05-W-Ol ^975^. 
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Expressive Writing; Selected Results From the Second National Assessment 
of Writing , Report no. 05-W-02 (1976). 

Explanatory and Persuasive Letter Writing , Report no. 05-W--p3 (1977). 

Write/Rewrite: An Assessment of Revision Skills , Report no. 05-W-04 
(1977) . 

Objectives 

A description of the 1973-74 and 1978-79 writing objectives and the 

procedures used in developing the objectives are available in Writing 
Objectives, Second Assessment (1972) . 

Exercises, Scoring Procedures 
and Data 

For those wishing to use specific National Assessment items. National 
Assessment provides copies of released items, exercise documentation and 
scoring guides as well as information on scoring procedures and exercise-level 
data in microfiche. Materials that are or will be available include: 

Ihe Third Assessment pf- Writing, 1978-79, Released Exercise Set (1980) — 
this loose-leaf set provides copies of all released ^vriting exercises 
from the 1978-79 writing assessment, detailed scojring guides, exercise 
docunentation (including timing and objective measured) and sample 
papers representing the various score points. Included with this set 
are the writing objectives and tv*o papers, described below, which 
explain National Assessment's procedures for scoring essays. 

Using the Primary Trait System for Evaluating Writing (Mallis, 1980) 
describes the rationale behind the primary trait system and how the 
system is applied. 

Guidelines for Describing Other Aspects of Writing: Syntax, Cohesion and 
Mechanics (Mil lis and Mellon, 1980) explains how syntax, cohesion and 
mechanics were scored in 1978-79 and presents summaries of the scoring 
guides used. 

Data Appendix for the 1978-79 Writinc^ Released Exercise Set (1981) — 
this microfiched appendix will provide data on the exercises included 
in the Released Exercise Set . For each exercise, percentages of 
respondents answering in various response categories will be presented 
for the nation and for designated popiiLation groups. 

For those desiring additional' exercises, the exercises released following 
the second assessment are available in Writing Exercise Set (1976). Data for 
these items are found in Report no. 05HV-20, The Second National Assessment of 
Writing, New and Reassessed Exercises With Ttechnical Information and Data 
(1978) . ^ 
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Pub-lic-Use Data Tape 

For tJiose who wish to perform their own analyses of National Assessment 
data^ National Assessment will make available a data tape of respondent-level 
data for the 1978-79 writing assessment, lb protect the confidentiality of 
the respondents, all identifying information (school, district, state) has 
been deleted. The tape includes docunentation of exercises and is organized 
and documented in such a way that it can be used with standard statistical 
packages. 



User Services 

National ' ^Assessment provides some assistance to those wishing to use 
assessment items or to replicate assessment methodology. 'Those interested in 
receiving assistance should contact the National Assessment office. 
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APPENDIX A 

DEFINITIONS OF NATIONAL ASSESSMENT REPORTING GROUPS 



In addition to reporting results for all 9-, 13- and 17-year^old students 
in the Lfriited States, National Assessment reports results for a nunber of 
population subgroups. Most of these subgroups were defined for all three 
writing assessments • 



The country has been divided into four regions: Northeast, Southeast, 
Central and Vfest. States included in each region are shown on the followina 
map. 



Reporting Variables 



Region 




Sex 



Results are reported for inales and females. 



Race 



Results are presented for blacks and whites. 
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Level of te rental Eaucation 

Three categories of parental-education levels are defined by National 
Assessment, based on students' reports. Ihese categories are: (1) those whose 
parents did not graduate from high school^ (2) those who have at least one 
parent who graduated from high school and (3) those who have at least one 
parent who has had some post high school education. 



Type of Community 

Three extreme^'cbmmunity types of special interest are defined by an 
occupational profile of the area served by a school as well as by the size oi 
the canmunity in which the school is located. 

Advantaged-urban (high-metro) communities . Students in this group attend 
schools in or around cities having a population greater than 200,000 v^ere a 
high proportion of the residents are in professional or managerial positions. 

Disadvantaqed-urban (low-metro) communities . St ud ents in thi s g roup 
attend schools in or around cities having a population .greater than 200,000 
where a relatively high proportion of the residents are on welfare or are not 
regularly employed. 

Rural communities . Students in this group attend schools in areas with a 
population under 10,000 where many of the residents are farmers or farm 
workers. 

This is the only reporting , category that excludes a large nunber of 
respondents. About two-thirds do not fall into the classifications listed 
above. Results for the remaining two- thirds were not reported since their 
performance was similar to that of the nation. 



Size of Community 

Big cities . . Students in this group attend schools within the city limits 
of cities having a 1970 census population over 200,000. 

Fringes around big cities . Students in this group attend schools within 
metropolitan areas (1970 U.S. Bureau of the Census urbanized areas) served by 
cities having a population greater than 200,000, but outside the city limits. 

^4dilJ^ cities . Students in this group attend schools in cities having a 
population between 25,000 and 200,000, not classified in the 
fringes-around-big-cities category. 

Small places . St ud ents i n thi s g roup attend school s in commun i t i es 
having a population less than 25, 000, not classified in the 
fringes-around-big-cities category. 
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Grade in School 



Results are categorized for 9-year-olds in the 3rd or 4th grade, 
13^ear-olds in the 7th or 8th grade, and 17-year-olds in the 10th, 11th or 
12th grade. 



Modal Grade by Region 

Results are categorized for 9-, 13- and 17-year-old respondents in grades 
•5.' 8 and 11, respectively, who live in the NortheasLern, Southeastern, Central 
or Western regions of the country. 



Modal Grade by Community Size 

Results are categorized for 9-, 13- and 17-year-old respondents in grades 
4, 8 and 11,* respectively, vs^o live in big cities^ fringes around big cities, 
meditn cities and small places. 



Modal Grade by Sex 

Results are categorized for 9-, 13- arid 17-year-old males and females in 
grades 4, 8 and 11, respectively. 



Instructional Experiences Related to Writing 

Thirteen- and 17-year-old students were asked a number of questions 
regarding their instructional experiences with writing. Responses to these 
questions were used in analyzing writing achievement data. The questions 
employed^ to gather information about experiences follow and are found in 
Appendix" B. These questions were given to 13^ear-olds in the 1978-79 
assessment and to 17-year-olds in both the 1973-74 and 1978-79 assessments^ 

Number of Papers Written: 13- and 17-Year-Olds 

Students were asked how many reports they had written in the last 6 weeks 
as part of any school assignment. Five levels were defined: (1) no reports, 
(2) 1 report, (3) 2-4 reports, (4) 5-10 reports and (5) more than 10 reports 
(used at age 17 in 1^73-74) . 

Instruction in Writing; 13- and 17-Year-Olds 

Students were asked about the time spent in their English 'classes on 
instruction in writing. Three levels were defined: (1) little or none of the 
time, (2) about one-third of the time and (3) half or most of the time (used 
at age 17 in 1973-74;. 
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Additional Remedial Writing Cburse: 17-Year^Ids 



Seventeen-year-olds were asked if they had taken an additional remedial 
writing course, TWo levels w&re defined: (1) yes and (2) no (used in 1973-74). 

Additional Writing Course Other Than Remedial Writing: 17-Year^lds 

Seventeen-year-olds were asked if they had taken an additional creative 
writing course or any other additional course besides remedial writing. TWo 
levels were defined: (1) yes and (2) no (used in 1973-74). 

Bicouraqed Prewriting: 13- and 17-'Year-01ds 

Students were asked if their teachers encouraged them to make notes 
before writing and if their teachers encouraged them to make outlines before 
writing. TWo levels were defined: (1) students v^io responded "usually" or 
"sometimes" to either or both questions and (2) students who responded "never" 
to both questions or who did not write papers. 

Draft or Rewrite Papers Before Hirning Inz 13- and 17-Year-Olds 

Students were asked if they wrote their papers more than once before 
turning them in to teachers. Three levels were defined: (1) students who 
responded "usually^" (2) students who responded, "sometimes" and (3) students 
who responded "never" or who did not write papers (used at age 17 in 1973-74). 

Teacher Provides Written Suggestions on Papers: 13- and 17-Year-Olds 

Students were asked if their returned papers had v^itten suggestions from 
their teachers. Three levels were defined: (1) students who responded 
"usually^" (2) students who responded "sometimes" and (3) students v^o 
responded "never" or had not written any papers (used at age 17 in 1973-74). 

Teacher Discusses Papers: 13- and 17-Year-Olds 

Students were asked if their teachers discussed their papers with them. 
Three levels were defined: (1) students v^o responded "usually^" (2) students 
who responded "sometimes" and (3) students who responded "never" or had not 
written any papers. 

Teacher Feedback: 13- and 17-Year-Olds 

Using .the information from students' responses about written suggestions 
from teachers and teacher discussions^ tw levels were defined: (1) students 
who responded "usually" or "sometimes" to either or both questions and (2) 
students :;ho responded "never" to both questions or had not written any 
papers. 

Work To Improve Papers After They Are Returned: 13- and 17-Year-Olds 

Students were asked if they worked to improve papers after teachers 
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reviewed t±iein. Three levels were defined: (1) students who responded 
"usually^" (2) students who responded "sometimes" and (3) students who 
responded "never" or did not write any papers (used at age 17 in 1973-74) • 

Good and Poor Writers: 9-f 13- and 17-Year-Olds 

' ■ ' y 

For exercises scored for mechanics and syntax^ three additional variables 
were analyzed: good and poor writers as defined by the (1) holistic, (2) 
primary trait cr (3) cohesion evaluations. For each variable, the categories 
were defined as follows: (1) levels 1 and 2 combined (poor) and (2). levels 3 
and 4 combined (good). These variables were only used in the anaJysis of 
syntax and mechanics characteristics* The variable(s) used was dependent on 
the type of scoring used in a particular exercise. 
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FORMS USED TO OBTAIN BACKGROUND INFORMATION 



This appendix includes the forms used by National Assessment to collect 
background information from school officials and respondencs for the 1978-79 
assessment. Following are a listing and a brief decription of the forms 
included. 

pp. 47-48 School Pr incipal ' s Questionna ire — filled out by school 
principals or other school officials for schools at each age 
level. 



p. 49 Supplementary Principal's Questionnaire — filled out by 

school officials from 13- and 17-year-olds' schools to provide 
information about school writing programs, teacher support and 
class size. (Complete questionnaire included items- about art, 
music and writing instruction.) 

p. 50 Package Cover Sheet — cover of item booklet filled out by 

exercise administratcrs to provide information about the 
grade, sex, birth date and race of each student* 

p. Directions to Exercise Administrators for Coding Package Cover 

Sheet — tells exercise administrators how to code information 
in boxes 3-8 on package cover sheet. Directions shov^ are for 
1 3-yea r-olds ' bookl ets . 

pp. 52-53 Standard Background Information Form for 9-Year-Olds ^ — 
provides information about reading material in the home and 
level of parents' education. 

pp. 54-55 Standard Background Information Form for 13-Year-Olds — 
provides information about reading material in the home, level 
of parents' education and place lived in at -age 9. 

pp. 56-59 Standard Background Information Form for 17-Year-Olds ~ 
.provides information on homework, TV watching, racial 
identification, ' possessions in the home and classroom 
activities, in addition to guestions asked of 9- and 
13-year-olds. 

pp. 60-61 :vyriting Background Questionnaire for 13-Year-Olds — provides 
information about f requency of wr i t i ng assignments , writing 
instruction and writing enjoyment. 
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pp. 62-63 Writing Background - Questionnaire for t7-Year-0Ids — provides 
information about frequency of writing assignments, additional 
writing courses, writing instruction and writing enjovment. 
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SCHOOL PRINCIPAL'S QUESTIONNAIRE 



This report is authorized by law (20 U.S.C. 1221 c-M. While 
you are not required to respond, your cooperation ts needed 
to make the results of this survey comprehensive, accurate, 
and timely. 



Primary Sampling Unit 



PLEASE 
PRINT 



School Number 
Age Group(s) 



13 



Name of School 

Address of School 



(Street) 



17 



(City) . 
Name of School Principal 



(State) 



(Zip Code) 



Name and title of person- completing the form if other than school principal 
Name Title 



!• What is your best estimate of jthe current enrollment and the average daily - 
■ attendance by grade of your school (1978-79 school year)? (Enter zeros for 
grades not served by your school.) 



Grade 


K 


1 


2 


3 


4 


5 


6' 


7 


8 


9" 


10 


11 


12 


Enrollment 


























Average 
Daily 

Attendance 




















1 









2. Approximately what percentage of the students attending your school live in each 
of the following areas? , 

% A In a rural area (less than 2,500) } 

' % B In a iiown of 2,500 to 10,000 * ^^^-^ 

% C In a. town of 10,000 or more 



100% 



(Ivems A-C should add to 100%) 
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Approximately what percentage of the students attending your school are childr 
of . ' 

% A Professional or managerial personnel 

% B Sales, clerical, technical or skilled workers 



_% C Factory or other blue collar workers 



% D Farm workers 



_% E Persons not regularly employed 



100% 



_Z F Persons on welfare 

(Items A-F should add "to 100%) 



Approximately what percentage of the students attending your school are 

% A .American Indian or Alaskan Vaciv. 

% B Asian or' Pacific Islander 



_% C Hispanic, regardless cf : ace .(Mexican, Puerto Rican, Cuban, Central 
or South American or ::ther Spanish culture or origin) 



_% D Black and not 'Hispanic 



1005S 



_% E White and not Hispanic 

(Items A-E should add to. 100%/ 



Does your school qualify for ESEA Title I assistance? 

Yes - If Yes, approximately what number of students qualify for 

and what number of students are receiving ESEA Title I assistance? 

Approximate number of students qualifying for ESEA 

Title I assistance 

Approximate number of students receiving ESEA Title I 

assistance 



No 



THANK YOU FOR YOUR COOPERATION 
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Supplementary Principal's Questionnaire 



Instructions; The purpose of this questionnaire is to provide additional, information which will 
be used in the analyses of NAEP data. Darken the appropriate ovals with a soft 
lead pencil. If you have questions about any of the following items, please co/itact 
the National Assessment district supervisor! Thank you for your cooperation. 

1. Does your school; 



Yes No 



A. 


Provide in-service training, instructional 
workshops, or professional improvement 
programs on how to teach writing? 




cz=> 


B. 


Have any form of individualized writing 
instruction for students, such as a 
writing lab, a remedial writing program, 
or writing tutors? 




<z=> 


C. 


Have an active departmental committee 
on writing? 







D. AiTsign fewer classes to teachers of 
writing? 



5. Provide assistants to writing teachers to 
help read student essays and papers (such 
as adults in your community, teachers in 
other areas in your school, teaching aids, 
or outstanding student writers)? 



2. Wh A is the average size of the language arts (general English, literature 
or grammar) classes in your school? 
cz) 0 to 12 students 



c=:> 13 to 18 students 
cz) 19 to 26 students 
(zz> 27 to 31 students 



czD 32 10 36 students 
CZD Over 36 students 



THANK VOU FOR VOL'R COOPKRATION 



School Name: 





IVS.r. and 




Sch(H)l Number 








/ ) 
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Example Package Cover Sheet 

PACKAGE NO. 



O.M.B. No. 0Si R1^ 
Aoorov«( E xpir*t 9-3a8l 



A 

8." 



.mm mm m- m 

m m m m m m 

m m * m 

« m mmmmm 

m m m m 

m m m m 

mmmmm mmmmm m m 





NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS 
A PROJECT Of THE EDUCATION COMMISSION OF THE STATES 



Th:s report is authorized by law (20 U.S C. -1221 e-1 ). While you are 
not required to respond, your cooperation is nectded to make the results 
f h;s survey comprehensive, accurate and timely. 











PACKAGE 








1 - 


1 






'ZD 










CD . 
8> 





16 



N 
P 

NR 
NE 





I. 


D. 






NUMBER 






I 














CD 






<=: 










(=> 






CD 




CD 



G5D 
OD 



□ 



□ 



* Grade 5 or lower c23 
'* Grade 9 or higher cZD 



YEAR 10 
AGE CLASS 2 
GROUP 



NCES Form Nos 2371 - !0. 57. 59 . 
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Di rections to Exercise Administrators 
for Coding Package Cover Siicet^ 



Codes To Be Used in Columns 3-8 

Package Admi. nistration 

Cod ing Schedul e 

C ol umn Column Item 

3 3 Grade 



Sex 



Bi r thda te 



Race 



EA Number 



8 — PSU and 

School 
Numbers 



Code 
Two digits: 

8th grade = 08 
Ung raded class = 98 
Spec i al educa t i o"n 
class = 99 

1 = Male 

2 = Female 

Month and last two 
digibs of year: 
May 1965 = 0565 

1 = W(Wh'ite) 

2 = B(Black) 

3 = S(Spanish Heritage) 

4 = I (American Indian 

or Alaskan Native) 

5 A( Asian or Pacific 
Islander) 

6 = U (Unclassi f ied) 

Two digits; number 
recorded on the front 
cover. of your manual 

Five digits; 

First two = PSU 

Number ; 
Last three = School 

Number ; 
as shown o-^ the 
Administration 
Schedule 
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standard Background Information Form for 9-Year-Olds 
1. Does your family get a newspaper regularly? 



2. Does your family j^et any ma^a/ines re^'ularly? 
CUD \vs c^D No cr::> I don't know. 

3. Are there more than 25 books in your home? 
CID Yes CZD No ( — ) I don't know. 

4. Is there an ent-yclopedia in your home? 

( — ) Yes CJD No ( — ) I don't know. 

5. How much school did your father complete? 

(FILL IN THK ON^F OVAL which best shows how much school your 
father completed.) 

(ZTD Did nnt complete the 8th ^rnde 

( ) Completed the 8th grade, but did not go to high school 
( ) Went to high school, but did not graduate from high school 
CUD Ciraduated fnjm high school ^ - 

CUD Some education after graduation from high school 






6. 



Did vour father graduate from a college or university? 



J Yes c__j No 



— -> I don't know. 
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7. 



8. 



How much schooM^id your mother complete? 

(FILL IN THE ONE oVaL which best shows how much school your 



CID Did not complete the 8th ^rade 

CZ) Completed the 8th grade, but did not go to high school 
CID Went to high school, but did not graduate from high school 
dD Graduated from high school 

CID Some education after graduation from high school 
CZ> I don't know. 

Did your mother graduate from a college or university? 
CID Vies cz:> No CZD I don't know. 
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standard Background Information Form for 13-YGar-Olds 



2 



3. 



4. 



5. 



6. 



Does your family j^et a newspaper regularly? 
CIZ) Yes c:^ No CID I don't know. 

Does your family <iet any magazines regularly*^ 
CU Yes ci\j No d'D I don't know. 
■ Are there more than 25 books in your home? 
CZD Vei; CZD No CZD 1 don't know. 

Ls there an encyclopedia in > our home? 
CZD Ves c ) No ( ) I don't know. 
How much school did youf" father complete? 

(FILL IN THP^ ONK OVAL >vhich best shows how much school your 
father completed.) 

CZD Did not complete the 8th grade ^ 
. CUD Completed the 8th grade, but did not go to high school 
CZD Went to high school, but did not graduate from high school 
CZD Graduated from high school 

CZD Some education after graduation from high school 
C_Z) I don't know. 

Did y(nir father graduate from a college or university? 
CZD Yes ciZ) No CZ; I don't know. 
How much school did your mother complete? 

(P'ILL IN THK ONL OVAL which best shows how much school your 

mother coti pleted.) 

CZ:^ Did noi complete the 8th grade 

CO Completed the 8th grade, but did not go to high school 
CZZ> Went to high school, but did not graduate from high school 
CZD Graduated from high school 

CZ) Some education after graduation from high school 
CZD I don't know. 

Did your mother graduate from a college or university? 
CZD Ves No CZD Idon^tkfiow. 
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9. Where did you live on your ninth birthday? 

0 In the I'nited States ( Please specify the stale or territory.) 



C-J Outside the L/nited States (Please specify the country.) 



c_ J I don't know. 
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standard Background Information Form 'for 17-Year-Olds 



A. How much school did your father complete? 

(FILL IN THK OVAL which best shows how much school your 

father com[)leted.) ^ ' * ^ 

CIO Did not complete the 8th Krade 

(LID (ompleted the 8th ^rade. hut did not to hi^rh school 
CZD Went to hi^h school, but did not graduate fn.m hi^h sch(X)I 
CZD (Iraduated from hi^h sch(K)l 

ciD Some education after Kr^duation from hiKh school 

CT-O I don't know. 

H. Did your father graduate from a collect* or university? 

'-^^ Ves No (Z-D I don't know. 

(*. How much school did your mother complete? 

(FILL IN THF ( )\F OVAL which best shows how much school your 
mother completed.) 
CLD Did not complete the 8th jjrade 

CZD ('om[>leted the Sth >rrade. but di(] not jjto to hiRh school 
CZD Went to hi^h schooL but did not >^raduate from hijrh school 
cr^ <^ira(iTaate<r from hi^h school 

dZ) Some educaiion after g-raduation from hi^2:h school 

ciD I don't know. 
D. Did your mother graduate from a college or university? 

czz> Yes No c — ) J don't know. 

K, Where di(i you live on your ninth birthday? 

CZD In the I'r^ited States (Please specify the state or territory.) 



c_Z) Outside of the Tnited States (L.. ase specify the country.) 



I don't know. 
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Where did you live on your thirteenth birthday? 
c"":) In the Tnited States (Please specify the state or territory.) 

CZD Outsitie of the Tnitecl States lIMease specify the country.) 

ciD I d<)n't know. 



Which (^f th(^ fc I lowing dors your family have at home? I Fill in one oval on 
^•ach line.) 

flave Do not have 



A. Nf\vs[)a|)er received 'v^ularly 

B. Ma^a/.i nes rrcei vt'd re>rularlv 



r. Mnff than 2^) Ix.ok: 
I>. Kncyclof^edia 



H. I>ictionarv 



Record plavtM 



Tape rvri,r(ivr or cassette player 



H. Ty pevvritt^ 



I. V'acuurTi cleaner 



.1. KIt'ctric <lish washer 



K. Two or more c^irs or trucks that run 



How miich tinu' did you s[)t»nd on homework yesterday? 

CIJ .\o fiorru*uork was assi>rned 
C'_j I h'ad honu'work [)ut didn't do it 
czj than one hour 

lii'tv.f t'n 1 and 2 hfjurs 

Mon* than 2 hours 



CZD 
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How many different scliools have you attended since you started the first 
jJTrade? 

dD 1 to '^ schools 
I to K schools 
c — ) 7 to 9 schools 
czz) 10 or more schools 

How lonp: have >*ou lived in the community in which you now live? 
CZ) All my life 

CZD 10 or more years but not all my life 

(ZD o to 9 years 

c — ) 2 to 4 years 

CZD 1 year 

czz) Less than 1 year 

How much television did you watch yesterday? 

CZD None (zz> 2 hours CD 5 hours 

czz) 1 hour or less czz) 3 hours cid 6 hours or more 

CZD 1 hour (ZD 4 hours 

Is Knj^lish the lan^uape spoken most often in your home? 
c — ) Yes czz) N'o 

Is a lanpTua^e other than Enjrlish spoken in your home? 

CZD Often (ZD Sometimes czd Never 

How many brothers or sisters do you have who are older than \ ou. 
None 12 3 4 5 6 or more 

CZD CD CZD CDD CZD CZD CZD 

How many brothers or sisters do you have who are younjrer than you? 



None 1 2 ^ 4 5 6 or more 

CZU (ZZ) -CZD CZD CZD CZD CZD 
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10. A. What is your racial backifround? 



American Indian or Alaskan Native 
Asian or Pacific Islander 
(=D Black. 
CZD White 

Other (Please specify) 



B. Is your ethnic heritage Hispanic (such as Mexican. Puerto Rican. 

Cuban. Central or South American or other Spanish Culture or origin)? 

czz> Yes czD No 

11. How often has each of the foilovving been used in the courses you are taking- 
this year? (Fill in one oval on each line.) 

Fairly 

Never Seldom Often PVequently 



A. 


■ Listeninfsr to the teacher's lecture c 




f s I 


B. 


Participatinjf in student-centered 
ciiscussions 


ZD CZZ) CZZD 


^ i 


€. 


Working: on a project or in a 
laboratory 


ZD CZZD CZZD 


j 

i 


D. 


Writing essays, themes, poetry, 

stories c: 


:D CZZD CZZD 


i 

i 


E. 


Goinj? on field trips ci 


Z) CZZD CZZD 


' czz^ : 


F. ' 




Having individualized instruction 

(small groups or one-to-one with 

a teacher) cz 


D CZZD CZZD 


1 

CZD 


G. 


Using teaching machines r .omputer- 
assisted instruction cz: 


D CZZD CZD 


CZD 


H. 


Watching television lectures cz 


D CZZD CZZD 


CZZ) i 


I. 


Studying from textbooks cz 


D CZZD CZZD 


CZZD j 


J. 


Library or media center assignments cz 


D CZZ) CZZD 


CZZ) • 
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Writinq Bcickqround Questionnaire for 13-Year-Olds 

Witw many rfp^rts and os>ays hnw you writtiMi during the last six 
■Af. ^.s as part of any srhodi a>si^^rHru'nt? 

In \hc \:vnrr:i\ Kny:\\>h. 1 i I cr'ani iv- or vrrarnniar issrs yuu havo takt'n 
ii'irinvr t tu' pa>t tuo >'L'ars, about wtiat part of ihv class time was spent 
or: iri>iruction in how to \\ rile re[)orts and essays? 
C^:^ Noru* t)f the time 

c o I.i 1 1 Ir of t fte I i r>u* ■ o ' - 

c_ Ahoiu onr-third (jif trie tir.ie 

Ahnut ont'-half of tht' 

y\(*>\ of ihv ti:..r 

A ro \-ou mcMii ravred to jot down ideas and niakt* notes about the to{)ic of 
;/M!;r pap'O* f)rf(o*e you write it? 

.l'-uai!\' SnmetirTies \e\-er I haven't written any papers. 

____ 

- -^J CZ) CZD CZD 

A vi r; 'U ravred l ^ niakr outlines of you r papers before you write 

' -^i:^'!;." Sor^iri I nu-s Never I hjiven't u'ritten an\- papers. 

'^.O (ZTD CZD 



PLEASE CONTINUE ON THE NEXT PAGE 
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E. Do you write a paper more than once before you turn it in to your 
teachers? 

Tsually ^^ometimes Never I haven't written any papers. 



F. When your papers are returned, do they have written suggestions on 
how to improve youi" writing? 

Usually Sometimes Never I haven't written any papers. 



(J. When your paper> are returned, do your teachers discuss them wMth 
you? 

Usually Sometimes Never I haven't written any papers. 



H. After your papers are returned, do you work on the paper agrain to 
improve it? 

Usually Sometimes Never I haven't written any papers. 



I. Do you enjoy working: on writing: assignments? 

Usually Sometimes Never • haven't written any papers. 




DO NOT CONTINUH 
l-NTIL TOLD TO DO SO. 
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Writing Background Questionnaire for 17-Year-Olds 



How many reports and essays have you written during the last six- 
weeks as part of any school assign nientV 



In the general En^rlish. literature or ^rrammar classes ,)u have taken 
(luring the past two years, about what part of the class time was spent 
on instruction in how to write reports and essa.vs? 

CZD Xone of the time 

CZD Little of the time 

About one-third of the time 

CID Al)(>ut one-half of the time 

<zz; Most of the time 



In addition lo the general English, literature or jrramm.ar classes you 
have taken during the last two years, have you had or are you now 
taking any of the followin^r cour ses concerned with how to"w»-ite? 



1. Creative writing course 

2. Remedial writing course 

.S. Other Writing? course 
(If other, please specify. ) 



1 es Xo I don't know. 
CO CZD CID 



A re you encoura^^ed to jot down ideas and make notes about the topic of 
your paper before you write it? - ' 

Tsually Sometimes Never I haven't written anv papers. 

CTD CZD CZZ^. CZD ' 



Are you encouraged to make ouL nes of vour papers before you write 
them? 



Usually Sometin^es Never I haven^t written any papers. 



PLEASE CONTINUE ON THE NEXT PAGE 
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F. Do you write a i)a{)t*r riiore than onre licfore you tin*n it in to your 
teache r<? 

I'suaily Sometimes Never I fun^Ti't written any papi'rs. 

CZD C3 C_J CZJ 



(i. When your pH{)ers are returned, (io they hnw written >ii^*^('st inns on 
how to irTii)rove your writing*' 

Tsually Sometimes Never I fiavtMrt written any pa[)ers, 
C_D- CZZ) C13 C-:_> 



ir When your pa[)ers are returned, do your teachers discuss tficm with 
you? 

I'suaily Sometimes Never I fuiv^^n't writton any papors. 



I. After your p)apers are returned, do you wor'k (.»n the paper aK'ain to 
im{)rove it? 

I'suaily Sometimes Never I haven't writton any pap(*rs. 



J. Do you enjoy working t>n u ritin^ assivrnrticnts? 

Tsually Sometimes Svvvr I haven't u ritten any papers. 



Questions 4, 5, 8, 9 and 10 were not administered in 1974; 
h 3ver, the remaining questions v;ere asked in both the 
1973-74 and 197a-79 assessments . 




STOP 



DO \()| ( ()\ I l\l I 

I \ 1 1; loi I) K) DO so. 
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APPENDIX C 



RESPONSE RATES FOR ASSESSMENT SAMPLES 



Table C-1 shows the response rates for students assessed in 1969-70, 
1973-74 and 1978-79. In the 1969-70 and 1973-74 assessments the same general 
procedures were used to insure sample coverage. For each of the three age 
groups, 12 students and 4 alternates were selected for each assessment 
session. If all 12 students appeared for the session, then the alternates were 
dismissed. Otherwise, enough alternates were selected to bring the size of the 
group to as many students as possible up to a total of 12. If the group 
assessed nunbered b'>tween 8 and 12 students, then the administration was 
considered canplete If the final total was not at least a quorun of 8, a 
second and sometimes a third make-up session was held. The percentages in 
T^ble C-1 are based on the numbers of students assessed from the original 
groups of 12 selected and do not reflect the use of alternates. 

• For the 1978-79. assessment, slightly different procedures were used. The 
nuriber of students selected for each administrative session varied from 16-25 
students depending on previous response rates obtained from schools in similar 
communities. No alternates were selected. The quorun size needed to consider 
an administrative session complete varied according to the nimber of students 
selected. Since nonresponse rates have always been relatively small for ag.es 9 
and 13, the make-up or follow-up procedures used in 1978-79 for those ages 
were similar to the ones used for the first two writing assessments. If a 
quorun was not obtained at the first administrative session, a second and 
sometimes a third make-up session was held. At age 17, in the 1978-79 
assessment, follow-up procedures were conducted on a school, rather than a 
session, basis. If a school had an overall response rate of less than 75%, 
then all nonrespondents in the school were contacted for one or two follow-up 
sessions. These follow-ijp procedures for 17-year-oJds provided sample coverage 
similar to that obtained at ages 9 and 13. 

Since response rates at age 17 have always been somev*iat lower than at 
the other two ages, the Research Triangle Institute (RTI), Raleigh, North 
Carolina, was asked to conduct a special study of nonrespondents during the 
1972-73 assessment of science and mathematics. The results (Kalsbeek et al.^ 
1975; Rogers et al., 1977) indicated that about 80% of the total nonrespondent 
group did not appear at the assessment sessions because of conflicting school 
activities or illness. The renaining 20% did not seem to be available. They 
attended school infrequently, if at all (for practical purposes, they had 
dropped out), or they had moved out of the school attendance area. In either 
case, these students probably should not have been listed in th3 in-school 
population of eligibles. 
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TABLE C-1. Mjmber of Students Assessed and Response Rates, 
by ^e and Assessment Year 



Year 



1969-70 



1973-74 



1978-79 





iype 




10 tiai 


Average 


Response 






or 


Numoer 


Number 


Rates 




/luiTiiniS"" 


Packages 


of 


Assessed 


in 








Students 


Per 


Percent 








Assessed 


Package 




Q 

y 




8 


19f 468 


2,434 


88.0 




r 
1 




0/ 713 


1,856 


89. 1 


1 J 








2,411 


85. 6 




T 




D f boo 


l,o5o 


87. 2 


1 7 


o 






z, UoJ 


74 » 5 




I 


2 


3,328 


1,664 


71.2 


Q 




iU 


Zbf 503 


2, 650 


90.4 




T 
i 


1 


2, 233 


2, 233 


89. 1 




o 


1 ^ 

IH 


JO, UoO 


2, 577 


86. 5 




I 


1 


2,239 


2,239 


88.6 


17 


G 


16 


36,709 


2,294 


75.2 




I 


1 


2,163 


2,163 


79.3 


9 


G 


11 


27,605 


2,510 


87.8 


13 


G 


13 


35,910 


2,762 


84.9 


17 


G 


14 


37,822 


2,702 


81.4 



» • V— A 

"*"In the 1969-70 and 1973-74 assessments, some booklets were administered to 
individual students using an interview mode (I). Ihe other booklets were 
administered to groups of students (G). 



Tobies published in previous National Assessment reports showing response 
rates for age 17 generally contain percentages adjusted to account for those 
17-year-olds listed, but not attending school. But, since National Assessment 
has not had the resources to replicate the RTI study in recent assessments, 
the 20% figure used as a basis for adjusting these percentages may be outdated 
and thus, the percentages given in 'feble C-1 have not been adjusted. It seems 
likely that despite efforts to update the lists of eligibles, these lists 
still contain some percentage of students who have in effect left the schools. 
Thus, the percentages listed for age 17 are probably underestimates of the 
actual response rates for 17-year-olds attending school. 
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APPENDIX D 



COMPUTATION OF MEASURES OF ACHIEVEMENT, 
CHANGES IN ACHIEVEMENT AND STANDARD ERRORS 



Several measures of achievement that National Assessment uses in its 
reports are described in Chapter 7 of this document. The sample design, as 
described in Chapter 4, is a complex, deeply stratified, multistage 
probability sample design* Measures of achievement are obtain^c by weighting 
individual responses appropriately. Reasonably good approxir.ation of standard 
error estimates of these achievement measures can be obt-;^ined by applying hne 
jackknife procedure to first-stage sampling units (FSUs^ within strata, using 
the method of successive differences and accumulating c^jross strata. 

In this section, the measures of achievement are first defined in 
algebraic form, follovs^d by a description of the jackknife method that 
National Assessment uses to estimate the standard errors of achievement 
measures. 

Measures of Achievement 

. Based on the sample design, a weight is assigned to every individual vtio 
responds to an exercise administered in an assessment. The weight is the 
reciprocal of the probability of selecting a particular individual to take a 
particular exercise with adjustment for nonresponse. Since the probabilities 
of selection are based on an estimated number of people in the target age 
population, the weight for an individual estimates the nunber of similar 
people that individual represents in the age population. As explained in 
Appendix ^, the weights- are adjusted to reflect information from previous 
assessments on population distributions. 



A sum of the weights for all individuals at an age level responding to an 
exercise is an estimate of the total nunber of people in that age population. 
A sum of weights for all individuals at an age responding correctly to an 
exercise is an estimate of the nunber of people in the age population vitio 
wDuId be able to respond correctly if the entire population were assessed. 
These concepts also apply to any reporting group (e.g., defined by region, 
sex, and so on) and category of response (e.g., correct, incorrect and "I 
don't know") . 

e ^ 

Let ^^Yih ~ weights for respondents to exercise e who are in 

^ reporting subgroup i and' who are in the kth replicate of the 

hth sampling stratum, and 
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ihk " °^ weights for respondents to exercise e who are in 

reporting subgroup who are in the kth FSU of the hth 
sampling stratun and who selected response category j (e.g., 
correct foil) for the exercise. 



Note that = E c^^ 

Ihk . ihk 

Ihen summing k over the sample FSUs in the stratm h, and summing 



over the H sampling strata. 



H 

1++ ihk 
h=l k=l 



estimates the number of 



eligibles in the population who are in subgroup i* 

H 

Similarly, / ^ihk 

h-1 k=l 

estimates the nunber of eligibles in the population who are in subgroup i and 
who WDuld select response category j for exercise e. 

An estimate of the proportion of the eligibles in the age population in 
group i v^o would select response category j on exercise e is: 



(1) 



In the special case where the proportion of all age eligibles who would 
select response category j on exercise e is estimated, the index A (for ALL) 
will be used in place of i as follows: 



(2) 



In National Assessment reports, the proportion in (1) • -iplied by 100 
is called the group percentage, and the proportion in (2) multiplied by 100 is 
called the national percentage. The difference between the proportion in 
subgroup i who vrould select category j on exercise e and the proportion in the 
nation is denoted by: 



(3) iP^^ 



P«3 



National Assessment also reports the arithmetic mean of the percentage of 
correct responses over sets of exercises corresponding to the measures in (1), 
(2) and (3). These means are taken over the set of all. exercises or a subset 
of exercises classified by a reporting topic or content objective. The mean 
percentages of correct responses taken over m" exercises in some set of 
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rcises corresponding to measures (1), (2) and (3) are, respectively: 



(4) P. * - r I , 



(5) p, = - : c^^^ / 

^ ^ A-^+ 1+^. ana 



Note that response category subscript i has been suppressed si nee the 
means are understood to be taken over the correct resoonse category for each 
exercise. 



Each of these six achievement measures is corr.puted and routinely used in 
reports describing achievement data for any assessment. Ihe simple difference 
m these measures between tWD assessments of the same exercise (or sets of 
exercises) provides six measures of change in achievement that are routinely 
used m National Assessment's change reports. Ihe next section describes how ^ 
standard errors are estimated for the 12 statistics used in NAEP reports. 



Computation of 
Standard Errors 



In-order to obtain an approximate measure of the sampling variability in 
the statistics (1) through (6), a jackknife replication procedure for 
estimating the sampling variance of nonlinear statistics from complex, 
multistage samples w«as tailored to National Assessment's sample design. 
Miller (1968, 1974) and Mosteller and TUkey (1977) provide information about 
the jackknife technique, while Folsom (1977) describes how the procedure is 
used in estimating standard errors for National Assessment's sample "design. 

_ To demonstrate the computational aspects of this technique, consider 
estimating the variance of the statistic in (1) — the proportion of 
age-el igibles in subgroup i who WDuld select response category j on exercise 

TTiis statistic is based on the data from all the n^ FSUs in the H strata. 

^i-hk defined- as a replication estimate of p^^ and constructed from 
ail the replicates excluding the data from FSU k in stratun h. Tliese 
replication estimates are computed as if the excluded FSU had not responded. 
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and a reasonable nonresponse adjustment is used to replace the data in FSU hk 
in estimating P^-^ . Several choices for replacing the data in FSU hk are 
available. In order to obtain a convenient and computationally efficient 

algorithm for approximating standard errors. National Assessment replaces 

ei e 

^ihk ^ihk hkth FSU with corresponding sums from another paired FSU 

in the saiT ^tratun. The replicate escipnate is then ccxnpute^. The replicate 
estimates to be used in the calculati^Dns are determined by arranging all the 

FSUs in each stratum into successive^ pa rs. That is, FSU 1 is paired with FSU 

i 

2, FSU 2 with FSU 3, 3 wi th 4, . . . -"^"^^ with and FSU n^ with FSU 1. 

ei 

The contribution to the variance of P, by each pair of FSUs is the 
change in the value of the statistic incurred by replacing the data from each 

FSU in the pair with the data from the other FSU in the pair and reconputing 

e i 

P^-^ in the usual way, This produces tw replicate estimates. Squaring the 

difference between these replicate estimates and then dividing by eight 

measures the contribution of this pair of FSUs to the total variance. The sum 

of these contributions over all n^ successive pairs in the stratun is the 

contribution by stratum h to the total variance. The sq:iare root of the sum of 

the H stratuTi contributions is the estimate of the standard error of p^-^ . 

1 

Algebraically, the tWD replicate estimates for tlie pair k, k+1 {^ere 
k=l,,».n^ and-n^+l=l) are: 

.-ej _ ^ej ^ ej 

(7) P®^ = ihk ih(k-Kl) 

i-h:< ej _ ej ej 

"^ihk ^ih (k-^I) 

and 

(8) p®^ - i-^-^ ^ih(k-H) ^ihk 

i-h{k-^l) ^ej _ ej ej . 
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The contribution to the total variance from stratu- h is: 



(9) var CP^^) = ^ 



.^d finally, an estimate of the standard error of .^^^ is: 



H 55 
<^Z) SS (Pf^) = C var P^c^ . 



Multiplying p^^ by 100 yields the percentage of response to category 
Muitipiyir>g se:(?^^) by 100 yields the corresponding estimated standard 
error of the percentage. 

In general, the jackknifed standard errors of the proportion. estimates 

will be larger than the simple random sampling formula (PQ/N)'\ where 
_ e j 

P-P^ , Q=l-P and N is the number of sampled respondents in subgroup i who took the 
exercise. The larger size of SE (p^^) reflects mainly the loss of precision 
dup to cluster-sampling of schools and students. Tiie standard errors for the 
achievanent measures (2) through (6) are computed through a series of steps 

analogous to those followed in computing SE (P^^) . 

i 

The standard errors for the differences between two assessments for any 
of the achievement measures (1) through (6) are computed as the square root of 
the sum of the squared standard errors from each of the separate assessments. 

The size of the standard errors depends largely not only on the nunber of 
FSUs and schools included in the sample, but also on the nunber of respondents 
in each of the reporting groups. T^ble D-1 shows the average nunber of 
students responding to an exercise booklet for each of the reporting groups 
for each age for each assessment year. T^ble D-2 shows Natior.al Assessment's 
current estimates of the proportions of students in each reporti,-in qroup at 
each age. - ^ c- 
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TABLE D-1. Average 
Takinq an Item 



Number of Respondents in Reporting Groups 
Booklet, by Age and Assessment year+ 



A'JO 9 At' 13 

1969-70 1973-74 1978-79 1969-70 1973-74 1978-79 



1969-70 1973-74 





2,142 


2,614 


2,510 


2,226 


2,556 


2,762 


1,994 


2,301 


2,702 


Roq i '^n 






















546 


621 


580 


584 


626 


675 


557 


549 


642 




435 


634 


625 


523 


625 


657 


406 


553 


683 


^ ♦Tit r .1 1 


509 


751 


665 


533 


718 


752 


490 


673 


725 




60 1 


608 


639 


585 


587 


681 


540 


524 


649 


Mil I'- 


1 f U / J 


1 111 




1,081 


1 , 297 


1,370 


970 


1,127 


1,312 


Kfi?,-! 1. • 


1,068 


1,301 


1,255 


1,129 


1,259 


1,395 


1,024 


] , 174 


J , 386 


Wh it" 


i f V V 


9 n ^ 1 

^ , U D i 


1 Q i1 Q 
i. / 0 *1 7 


1 L C C 

1 , boo 


^,03 


2,053 


1,649 


1,907 


2,134 




i 


4 ] 7 


A Q A 


Jo6 


393 


507 


230 


295 


392 




1 7 J 


i 4 c 


1 7 7 


i /4 


132 


202 


115 


99 


176 






















. J r 'Kiil.T t 'Ml (1 icjn 






















^ J 7 


974 


1 7 L 


^ "3 C 




326 


417 


447 


386 






'585 


589 


712 


766 


895 


657 


745 


^95 


F*os t hi^fi school 


7ft ] 


7 7 7 


ft fin 


Q 1 


T Art 1 

i , 002 


1, 200 


866 


1,021 


J ,334 


I inknowfj'*' 


V ^ L 


70 J 


0 jU 


T T T 
ill 


399 


341 


32 


88 


87 


Tvtf * ''if ''f •niTiii n 1 1 V 




















Kxt. r'"^m" rur-i 1 


216 


207 


252 


225 


245 


271 


203 


236 


246 


I'l.T'l 1* 'III' (141 l^lt <Jl( 


til 




^ jL 


JO 


1 A A 

^99 


281 


195 


289 


306 




209 


20? 


^ J J 


? 1 n 




£. 1 1 


19 / 


210 


265 


f>t|ifr '"ornmu n i f i 


i. , J V 


1^071 


1 7 r 


1 5 1 
1 , 50 1 


I , d98 


1,933 


1,399 


1, 566 


1,885 






















Pi'I t i 's 


593 


670. 


723 


597 


717 


775 


536 


546 


718 


Krin';»T> jro'in^l biq 




















ci t i»-<^ 


349 


435 


448 


352 


399 


588 


315 


340 


571 


,M*'<! i niTi r i t i ♦•s 


293 


365 


237 


346 


392 


287 


320 


349 


287 




905 


1,144 


1,097 


929 


1,048 


1,115 


823 


1,066 


1,123 


^ \ 10 


494 


j71 


623 


546 


707 


683 


256 


2y'* 


160 


4, 11 


i,559 


1,976 


1,834 


l,5ci5 


1,764 


2,03] 


1,379 


1,70:; 


2,014 


12 














219 


251 


285 




89 


67 


S3 


95 


85 


48 


140 


40 


43 



t .1 r iy ii'tf t • .U rill.' t.M rnijnn i i 
1 /-y»Mr-*'»lH? ^•nroll«"i in srhooj , 



TABLE D-2. Estimated Current Population Proportions 
of National Assessment Reporting Groups 
for In-School Students 



Reporting Groups 

Sex 

Male 
Female 

Race 
W". ite 
D±ack 
Other 

Region 
'"northeast 
Southeast 
Central 
West 

Parental education 

Not graduated high school 
Graduated high school 
Post high sclxjol 
Unknown 

Type of cominunity 
Rural 

Disadvantaged urban 
Advantaged urban 
Other communities 

Size of community 
Big cities 

Fringes around big cities 
Mediun cities 
Small' places 

Grade in school 
<3, <7, <10 

3, 7, .10 

4, 8, 11 
<4, <8, 12 
Other 



9 13 17 

.50 .50 .48 

.50 .50 .52 

.79 .80 .83 

.14 .13 .12 

.07 .07 .05 

.25 .25 .25 

.22 .23 .20 

.27. .27 .29 

.26 .25 .26 

.09 .13 .15 

.24 .32 .32 

.3? .42 .48 

.34 .13 .05 



.08 .10 .08 

.07 .07 .09 

.11 .11 .11 

.74 .72 .72 

.20 .21 .19 

.22 .22 .26 

.12 .11 .11 

.46 .46 .44 



<.01 .02 .02 

.23 .25 .13 

.75 .72 .75 

<.01 <.01 .10 

<.01 <.01 <.01 
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ADJUSTMENT OF RESPONDENT WEIGHTS BY SMOOTHING TO REDUCE 
RANDOM VARIABILITY OF ESTIMATED POPULATION PROPORTIONS 



Backg round 

As noted elsewhere, a weight is :issigned to every individual who responds 
to an exercise administered in an assessment. The weight is the reciprocal of 
the probability of selection of the individual with adjustment for 
nonresponse, and the probabilities of selection are based on the estimated 
nunber of people in the target age populations. Therefore/ the weight for an 
individual estimates the nunber of people that the individual represents in 
the age population* The sum c: the weights of all individuals at an age level 
who responded to an exercise is an estimate of the total nunber of people in 
that age population in the year that the exercise was assessed. Similarly, 
the sum of weights for all individuals v*io took the exercise and viho also are 
members of some demographic category (such as blacks) gives an estimate of the 
number of people in the aqe population, for the year, who are also members of 
the category. The ratio of the tvo totals estimates the proportional 
Lepresentation of the demographic category in the age population for the given 
year. 

Separate estimates of the proportional representation of the various 
demographic subgroups are provided by each booklet administered to a 
particular age group in a given year. Cue to random sampling variability, the 
estimates of population proportions for a given year based on single booklets 
will vary. There is also random sampling variation in ertimates of population 
proportions from year to year in addition to whatevei trends in population 
proportions over time that might exist. 

It is desirable to reduce the random variability of population 
proportions as much as. possible since this variability has an effect on 
performance estimates. For example, the percentaga of acceptable responses 
for an age group is a function of the relative proportions of high-performinrj 
and low-performing groups. If the relative proportions of these groups are 
very different in different assessments due to sampling variability, then a 
portion of the change in percentage of acceptable responses for an age group 
could oe attributable to yearly sampling differences in the relative 
proportions of high- and low-achieving groups. 

In addition to reporting performance estimated for an age group as a 
wnole. National Assessment ^^also reports performance for various 
subpopuldtions, such as whites or blacks. Because variability o:" subgroups 
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within t±ese subpopuiations (such as males and females within the white 
subpopulation) influence the performance estimates for the subpopuiations, it 
is desirable that fluctuations of proportions of all subgroups of each 
subpopulation be reduced as much as possible. 

For each age and year, each of the various booklets administered provides 
estirnates of a given population profx^rtion. Since these estimates are subject 
to bookiet-to-booklet variability, a better estimate of the population 
proportion, which will have reduced variability, is obtained by combining the 
information from all booklets. However, these proportions vary from year to 
year due to random sampling variability or systematic differences in sampling 
procedures. An even better estimate of population proportions for any single 
year can be obtained by smoothing the proportions over several assessment 
years. The word ''smoothing" is used here in the sense of fitting a smooth 
curve to a sequence of nunbers by robust/ resistant procedures (Tukey, 1977). 
Smoothing estimates of population proportions reduces a large portion of the 
sampling variability while preserving, as far as possible, actual trends 
occurring in the age population. 

After the population proportions have been smoothed, adjusted weights are 
derived for the assessed individuals so that the population proportions * 
computed using the adjusted weights are equal to the smoothed proportions. 
The adjusted weights are then used for all analyses. 



Smooth i ng Procedures Used by 
National Assessment 

Tne most direct way to smooth proportions is first to classify people 
into mutually exclusive multiway cells on the basis of their membership in 
categories of various important variables and then to smooth the proportions 
within each of the resulting multivay cells across years. lAifortunately, this 
procedure tends to produce a large number of cells with few people and, 
consequently, quite unstable estimates of smoothed proportions. 

To circumvent this difficulty. National Assessment has utilized various 
smoothing procedures. These procedures, which are all basically 
weighting-class adjustments applied independently to each age, are designed to 
control, to varying degrees, fluctuations in certain key subgroups while 
avoiding, as much as possible, instabilities due to snail cells. 

The procedure used in 1978-79 has the following characteristics: 

1. It produces a single adjusted weight for each individual. 

2. It affords good control on the distribution of proportions of certain 
key variables. 

3. It produces the greater stability of performance estimates than other 
procedures that have been used. 

4. It is relatively easy to implement • 
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Even though adjusted weights using this procedure differ slightly from 
the corresponding adjusted weights from the other procedures that have 
previously been employed. National Assessment intends to use weights obtained 
using ".he 1978-79 procedure for all future analyses of data assessed in 
earlier years. This is simply because we believe weights obtained through this 
procecjre to be the best available. ^ 



The Current Smoothing 

Procedure : 

Tne ^ first step in the 1978-79 smoothing procedure involved the 
partitioning of the population of age. class eligibles into the six smoothing 
cells given in T^ble E-1. The same cells were used for all ages^ 



TABLE E-1. anoothing Cells Used for the 
1978-79 Smoothing Procedure 



Cell 


Race 


Fiegion 


1 


'White 


All 


2 • 


White 


. All . 




Whiite 


All 


1 

*^ 


Black 


SE 


5 


Black 


Not SE 


6 


Other 


All 



Community Size (CS) 

Big city + fringe (BC+FR) 

Median city (MC)- 

Snail places (SP) 

All 

All 

AJl 



Then,- for each age and every year, the proportion of the .-population in 
^•ach of the cells was estimated. For a given age and year, ^the^* proportion of 
!ihe popul-:tion in a particular cell was computed as the sum of weights of all 
respondents assessed in the given year were of the specified age and who 

belonged in the cell, divided by the' tjtai of the weight of all respondents of 
' -e given age assessed in that year. 

Ea-h of the six cells was comprised of a sequence of estimated population 
:..roporL. - corresponding to the various years of assessment, iilach such 
5eq.;enr f proportions was then smoothed by fitting robust/resistant lines. 
j'^;:ng data fron the U.S. Census and Current Population Surveys, trends in 
v:. — Llr.er^n by jge and race and by age and region were obtained. The data from 
these s : ys \^re adjusted to correspond with N/'^EP definition: as much as 
vcsvr.lo. ^ res^star.l lines within the 53Pioothing cells were constrained to 
sitisf" -::.e -.rend fron the U.S. Census and Current Population Surveys data. 

■rh^ i^^^p : the smoothing procec:\jre was to adjust the respondents' 

^jl.^rts to :oe -onF::3tent with the smoothed proportions. Since each respondent 
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takes only one booklet^ the weight adjustments were done independently for 
each booklet. For a given age, year and booklet, population proportions using 
the original weights were obtained for each of the smoothing cells. Then the 
weights of all respondents of a given cell were multiplied by the ratio of the 
smoothed cell proportion to the proportion using the original weights. Tin's 
produced the adjusted weights that were used in all analyses. 

Adjustment of Weights 
by Users 

The smoothed population proportions for 9-, 13- and 17-year-olds 
(in-school only) are given in Tables E-2, E-3 and E-4, respectively. Hie 
colunns of each table represent the smoothing cells while the rows represent 
the assessment year. For example, the smoothed population proportion of 
9-year-olds in snoothing cell 2 (whites in medium cities) for 1972-73 is 
.1158. 

To adjust respondent weights to be consistent with the smoothed 
proportions, the following procedures were followed: 

1. For each booklet, respondents were classified according to smoothing 
cell, and the raw population proportions for each cell were obtained. 
For example, the raw proportions for a booklet given to 9-year-olds 
in smoothing cell 4 was the total of the weights of ell 9-year-olds 



TABLE E-2. Smoothed Frequencies From 10-Year Shiooth 
by Spnoothing Cell and Year for 9-Year-Olds 

Cell 





1 


2 


3 


4 




6 


Race 


Wiit*- 


Wiite 


.Wiite 


Black 




Other 


Region 


Ail 


All 


All 


SE 


I, SE 


All 


Size of 














Community 


BC+FR 


MC 


SP 


All 


AU 


All 


Year 














1969-70 


C.3293 


0. 1258 


0. 3546 


0.0S15 


0.0745 


0.0624 


1970-71 


0.3251 


0. 1224 


0,3601 


0.0547 


0.0746 


0.0631 


1971-72 


0.3210 


0.1191 


0.3656 


0.0558 


0.0747 


0.0639 


1972-73 


0.3168 


0.1158 


0.3711 


0.0570 


0.0748 


0.0646 


1973-74 


0.3126 


0. 1124 


0. 3766 


0.0581 


U.0748 


0.0654 


1974 -75 


0. 3085 


0. 1091 


0.3822 


0.0592 


0.0749 


0.0661 


1975-76 


0.3043 


0.1058 


0.3877 


0.0604 


0.075C 


0.0668 


1976-77 


0.3001 


0.1024 


0.3^32 


0.0615 


0.0751 


4 0.0676 


1977-78 


0.2959 


0.0991 


0. 987 


0.0627 


0.0752 


0.0683 


1978-79 


0.2918 


0.0958 


0.^. .2 


0.0638 


0.0753 


0.069] 
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TABLE Smoothed Frequencies Fran 10-Year anooth 

by anoothing Ctell and Year for 13-Year-Olds 

Cell 





1 


2 


3 


4 


5 


6 


Race 


V*iite 


Vtiite 


V*iite 


Black 


Black 


Other 


Region 


All 


All 


All 


SE 


Not SE 


All 


Size of 














Community 


BC+FR 


MC 


S? 


All 


All 


All 


Year 














1969-70 


0.3200 


0.1'i09 


0. 3703 


0.0513 


0.0673 


0.0602 


1970-71 


0.3201 


0.1256 


0.3731 


0.0521 


n.0681 


0.0610 


1971-72 


0.3202 


0.1202 


0.3760 


0.0528 


0.0690 


0.0618 


1972-73 


0.3203 


0.1149 


0.3788 


0.0536 


0.0698 


0.0626 


1973-74 


0.3203 


0.1096 


0.3816 


0.0544 


0.0714 


0.0642 


1974-75 


0.3205 


0.1043 


0.3845 


0.0552 


0.0714 


0.0642 


1975-76 


0.3206 


0.0989 


0.3873 


0.0560 


0.0722 


0. 0650 


1976-77 


0.3207 


0.0936 


0.3901 


0.0567 


0.0731 


0.0658 


1977-78 


0.3207 


0.0883 


0. 3929 


0.0575 


0.0739 


0.0666 


1978-79 


0.3208 


0.0830 


0. 3958 


0.0583 


0.0747 


0.0674 



TABLE E-4. Shioothed Frequencies From 10-Year Shiooth 
by anoothing Cell and Ye r for 17-Year-Olds 

Cell 





1 


2 


3 


4 


5 


6 


Race 


Wiite 


Wnite 


V*iite 


Black 


Black 


Other 


Region 


All 


All 


All 


SE 


Not SE 


All 


Size of 














Community 


BC+FR 


MC 


SP 


All 


All 


All 


Year 














1969-70 


0.3405 


0.1447 


0.3686 


0.0415 


0.0581 


0.0466 


1970-71- 


0.3419 


0. 1386 


0.3704 


0.0427 


0.0591 


0.0472 


1971-72 


0.3432 


0.1326 


0.3722 


0.0440 


0.060"" 


0.0478 


1972-73 


0.3446 


0.1265 


0.3740 


0.0452 


0.0612 


0.0484 


1973-74 


0.3460 


0.1204 


0.3759 


0.0465 


0.0622 


0.0491 


1974-75 


0.3474 


0.1143 


0.3777 


0.0477 


0.0633 


0.0497 


1975-76 


0.3487 


0.1082 


0.3795 


0.049C 


0.0643 


0. 0503 


1976-77 


0.3501 


C.1021 


0.3813 


0.0502 


O.OGdS 


0. 0509 


1977-78 


0.3515 


0.0961 


0.3831 


0.0515 


0.0664 


0.0515 


1978-79 


0.3529 


0,0900 


0. 3849 


•0.0527 


0.0674 


0.0522 
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in the booklet who were black and in the Southeastern regioHr divided 
by the total of the weights of all respondents to the booklet. 

"2. For each booklet and snoothing cell, a weig.ht adjustment factor as 
the ratio of the smoothed population proportion (for the appropriate 
^ger year and sanoothing cell)- over the raw population proportion was 
obtained. 

3. Ihe adjusted weights for an individual were the product of that 
individual's original weight and the appropriate adjustment factor. 

Changes in Smoothed 
Propo rt ions as New 
Assessments Are Completed 

rVery time an assessment is completed, a new time point is added to each 
of the sequences of population proportions within the smoothing cells. This 
means that, even though robust/resistant procedures are used, the addition of 
a new point may somewhat change the values of smoothed proportions for prior 
years* Additionally, any changes in methodology will impact the estimates. 

This means that the srtoothed proportions, with the additioh of the next 
assessment data, are apt to differ somevv^at from the corresponding smoothed 
proportions without the new data. Tstetional Assessment has adopted -he 
philosophy that the smoothed proportions, based on all currently available 
data using the best available algorithm, are the best available. Therefore, 
all subsequent analyses, for any year, will be done using this best available 
information, even though this may prodLice estimates that slightly differ from 
prior values. 
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GLOSSARY OF NATIONAL ASSESSMENT TERMS 



Ac ceptabl e response , Any response to an exercise that danonstrates 
achievement of the objective measured by that exercise. 

Administration time , Ihe total time allowed on the paced audio tape for an 
exercise. {Includes the time allov>^d for the stimulus and the response.) 

Administration tintetable . Time periods during the school year v*ien the 
' various age groups are assessed. The time periods are: 

October-December 13-^ear-olds 
Ja nua ry-^Febr ua ry 9-yea r-olds 

Ma r c h-Ma y 1 7-y e a r-o 1 d s 

Age group or age level . Three age groups have been sampled in all three 
writing assessments: 9-year-olds, 13-year-olds and 17-year-olds attending 
school. Birth date ranges for each age group in each of the three 
assessments are as follows: 

Assessment .. ^ Age 9 Aqe 13 Age 17 

1969-70 1960 1956 10/51-9/52 - 

1973-74 1964 i960 10/56-9/57 

1978-79 1969 1965 10/61-9/62 

Assessment . The docunentation of the progress in knowledge, . skills and 

attitudes of American youth. Measure, are taken at periodic intervals for 

each learning area, with the goal of determining trends and reporting the 
findings to the public and to the education community. 

Assessment administrator . Individual employed to administer: the assessment in 
p^.rticipating schools. ^ - ^ 

Background questions . Qjestions abcnt "respondents' instructional experiences 
with writing and frequency of the"r writing were included in writing item 
booklets. Stand, d background questions asked in every learning area are 
found: on the back pages of the item booklets and include such things as 
level- of parental ed'jcation and reading 'mater ials in the home. Background 
questions used in the 1978-79 writing assessment appear in Appendix B. 

Booklet. Items (exercises) are presented to respondents in booklets. Booklets 
are designed to be scored by optical scanning machines. Each booklet 
contains (1) instructions for answering items and sample items, (2) 
assessment items airi (3) background questions. Each booklet contains 
approxiinately 30-35 minutes of assessment itens and 10-15 minutes of 
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introductory material and background questions. 

Category (scoring) . A classification of a response to an open-ended exercise. 
See scoring guide . 

Category within a variable ^ A subclassif ication within a variable. For 
example^ male and female are categories of the variable sex. .M.so see 
reporting groups . 

Difficulty level . The percentage of acceptable responses to an exercise. 

Exercise . A task designed to measure an objective. Because NAEP does not 
administer "tests/' but instead descr-ibes. education achievment over time, 
the temi "exercise" is often used instead of the term "item" or "test 
item." Ihe terms "item" and "exercise" are used synonymously in this 
report. 



exercise part . See item part . 

Exercise pool . The entire set of exercises prepared for a learning area. This 
set includes recycled exercises, exercises developed for previous 
assessments but not used due to exercise booklet or budgetary constraints, 
and newly developed exercises. 

Field test . A pretest of exercises to obtain information regarding clarity, 
difficulty levels, timing, feasibility and special administrative problems 
needed for revision and selection of exercises to be used in the 
assessment. 

Grade in school . Results are reported for 9-year-olds in the 3rd and 4th' 
grades; 13-year-olds in the 7th and 8th grades; and 17-year-oids in the 
10th, nth and 12th grades. 

Group administration . Booklets were administered to groups of 10 to 25 
students in 1978-79. In 1969-70 and 1973-74, group size was 12 respondents. 
A paced audio tape was used to provide uniform instructions and oral 
presentation of exercises. 

Hand scoring (scoring) , The coding of responses in a format compatiblr^ with 
the optical scanning equipnent being used. Multiple-choice exercises can 
be directly machine scored;' however, responses to nen-ended exercises must 
be coded in scoring ovals so that they can then oe machine scored. See 
scoring guide . 

ID r.jnber . An identification number referring to the unique number assigned 
to each respondent. This number is assigned to preserve the anonymity of 
each respondent. NAEP does not keep records of the names of any 
ind ividuals. 

Item. See exercise. 



Exercise booklet. See booklet. 
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Item booklet. See booklet. 



Item part . Each part of an item that asks a separate question. F^rts may all 
pertain to one stimulus, such as a qrapn or a table, or may concern the 
same topic. 

Jackknife . T^e name of the algorithm used by NAEP to estimate standard errors 
of percentages and other statistics. 

Learni ng a rea . One of the areas assessed by National Assessment: 
reading/literature, writing, mathematics, science, citizenship/ social 
studies, art, music, career development. Also called "subject area." 

Level of parental education . These leyels are described in Appendix A. 

Modal grade . Hie grade in vs^ich the majority of each in-school age group is 
enrolled. For 9-year-olds, the mod3l grade is"' the 4th grade; for 
13^ear-olds, the 8th grade and for 17-year-olds, the 11th grade. 

Objective . A desirable education goal agreed upon by scholars in the field, 
educators and concerned lay persons, and established through the 
consensus approach. 

Objectives redevelopment . After the initial assessment of a learning area, 
one of the first steps in preparing for reassessment is a review of the 
learning-area objectives. This is carried out by scholars in the field, 
educators and concerned lay persons. These reviews may result in revision, 
modification or total rewriting of the learning-area objectives to reflect 
current curricular goals and emphases; they may also result in the 
endorsement of the objectives from the previous assessment as adequate for 
the next assessment. 

Open-ended exercise . A nonmulti pie-choice exercise that requires some type of 
written or oral response. 

Paced audio tape . A tape recording that accompanies each booklet to assure 
uniformity in administration. Instructions and exercises are played back 
from the tape recording so that reading difficulties will not interfere 
with an individual's ability to respond. Response time is included on the 
tape. 

Primary sampling unit (PSU) . First-stage sampling units, typically a county 
or a <jroup of contiguous counties. 

Principal's questionnaire . A data collection form given to school officials. 
The officials respond to questions concerning enrollments, size of the 
community, occupational composition of the community, and so forth. Samples 
of these quest ionna i res a re found in Append ix B. See also supplementary 
principal's questionnaire . 

PSU . See primary sampling unit . 
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Public-use d^ta tapes. Computer tapes containing ret. jndent-level exercise and 
background/demographic data and machine- readable documentation. These tapes 
are available for use by external researchers wishing tc do secondary 
analyses of National Assessment data. 

Racial/et hnic category . For the writing assesments, results are reported for 
whites and blacks. 

Receipt control. Procedures implemented by scoring staff to check in and 
screen materials from the field. Information gained from receipt control 
procedures is relayed to assessment administrative staff so that any errors 
may be corrected. 

Recycled exercises. Ihe set of exercises that is kept secure from one 
assessment to the next that will be used to measure change. (grov>^h, 
stability or decline) in performance for the learning area. 

Region. Che of four geographical regions used in gathering and reporting 
data: Northeast, Southeast, Central and W^st. States included in each 
region are shown in Appendix A. 

Released exercise. An exercise for which results and exercise text have been 
reported to the public. 

Released exercise set. A set of released exercises, including documentation 
and scoring guides, that can be purchased from National Assessment. 

Reporting groups. Cat -ories of variables for which National Assessment data 
are reported. Variable categories are defined in Appendix A. 

Rescore. :f an open-ended exercise was scored under different conditions than 
presently held or if passage of time might affect scoring, :esponses from a 
previous assessment may be rescored at the same time that responses from a 
later assessment are scored. Responses from an earlier assessment also may 
be held and not scored so that responses from a later assessment can be 
scored at the same time. 

Respondent. A person.-wb.o responds to the exercises in an assessment booklet. 

Response options. Different alternatives to a multiple-choice question that 
can be selected by the respondent. 

Review co nference . A conference held tc review the objectives of a learning 
area to assure their acceptance as measures of the objectives by scholars, 
educators and lay persons or to review exercises for racial, ethnic, social 
or regional bias. 

S ample . National Assessment does not assess an entire age population but 
rather selects a representative sample from the age group zo answer 
assessment items. (See Chapter 4 for a description of National Assessment 
sampling procedures.) 
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Scoring quide > A guide for hand scoring an open-ended exercise that specifies 
descriptive or diagnostic categories by giving definitions and sample 
responses. 

Scoring ovals . Scannable ovals prirtod beside Tiulti pie-choice options and 
printed at the bottom of the page for open-ended exercises (to oe used in 
handscoring) . When ovals are marked, they can be scored by machine and 
responses recorded by computer. 

Sex. Results are reported for males and females. 

Size of commanity . Results are reported for four size-of-community 
categories: big cities, fringes around big cities, medium cities and small 
places. These categories are defined in Appendix A. 

SMSA . Standard Metropolitan Statistical Area. SMSAs are economic and soci. 
units defined by the U.S. Bureau of the Census. 

Standard error . A measure of sampling variability for a statistic. Because of 
NAEP's complex sample design, standard errors are estimated by jackknifing 
first-stage sample estimates. 

Stem , The portion of an exercise that states the problem or asks the 
question. 

Stimulus , See stem^ 

Subject area . See learning area . 

Subpopulation or subgroup . Groups within the national population, such as 
males arr? females, for which data are reported. 

Supplementary pr incipal ' s questionnaire . A data collection fqfm given to 
school officials. On this form, officials are asked to respond to questions 
concerning course offerings, materials and staffing specific to the 
learning area being assessed. A sample of this questionnaire is found in 
Appendix B, See also principal's questionnaire , 

Ta pe script , ^ script prepared for the announ,.er to use in p'^oducing the paced 
tape. It indicates exactly what is to be read or not reaa and indicates the 
amount of response time allowed for each exercise. See paced audio tape . 

Timing . Most NAEP exercises are administered with a paced audio tape to 
standardize data collection conditions. The tape includes the amount of 
time students are allowed to respond to each exercise. 

Type of community . Results are reported for three type-of-community 
categories: disadvantaged urban, advantaged urban and rural. Definitions of 
thf?se catecories are found in Appendix A. 

User tape. See public-use data tape. 
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Variable ^. A classi ficacion of respondents, standard reporting variables are: 
region, sex, race, level of parental education, size of community, type of 
community and grade in school. 

Weight. A multiplicati- e factor equal to rhe reciprocal of the probability of 
a respondent being selected for assessment v.-ith adjustment for nonresponse 
?n estimate of the mimber of persons in the population represented by a 
respondent in ^-he sample, nieoreticaily, the sum of weights for all 
respondents at an age level is equal to the --umber of persons in the 
country at that age level. 



86 

0 



BIBLIOGRAPHY 



Srown, Rexford G. "Evaluations of Writing: Some Theoretical Considerations 
and Practical Suggestions." New York; Ford Foundation (forthcoming) . 

Chromy, J. and D. Horvitz. "Structure of Sampling and Weighting," 1969-70 
Science: National Results and Illustrations of Group Comparisons , Report 
no. 1, 1969-70 Assessment. Denver, Colo.; National Assessment of 
Educational E^cgress, EHucation Commission of the States, 1970. ERIC no. ED 
055 78^^. ISBN 0-89398-276-8. 

Final Report on National Assessment of Educational Progress Sampling and 
Wei ghting ^ctivLties for Assessment Year 10 . Research Triangle Park, N.C. : 
Research Triangle Institute, 1980. 

Final Report on Year 09 Field Operations and Data Collection Activities, 
National ^s-essnent of Educational Progress . Research Triangle Park, N.C: 
Research Triangle Institute, 1978. 

Final Report on Year 09 In-School Field Operations and Data Collection 
Activities, National Asessment of Educational Progress . Research Triangle 
Park, N.C: Research Triangle Institute, 1978. 

Folsom, Ralph E. National jg^sessment Approach to Sampling Error Estimation , 
Sampling Error Monograph 25U-796-5. Research Triangle Park, N.C: Research 
Triangle Institute, 1977. 

Kaisbeek, W.D. et al . No Show Analysis , final report. Raleigh, N.C: Research 
Triangle Institute, 1975. 

Miller, f.G. Jr. "A Trustvvorthy Jackknife," Annals of Mathematical Statistics ^ 
no. 1964. 

Miller, R.G. Jr. "Jackknifing Variances," Annals of Mathematical Statistics , 
no. 39, 1969. 

Miller, R.G. Jr. "The Jackknife — A Review," Biometrika , no. 61, 1974. 

Moor J, R. et al . T he National Assessment Approach to Sampling . Denver, Cblo.: 
National Assessment of Educational Progress, Education Commission of the 
States, 1^74. ERIC no. ED 099 416. 

Mocteller, F. and J-W. T\jkey. Deta Analyses and Regression . Reading, Mass.: 
.^dison-Wesley, 1977. 



87 



EKLC 



Mosteiier, F. and J.W. Tukey, "Data Analysis Including Statistics," in 
Handbook of Social Psychology ^ 2nd ed., edited by E. Ar onsen and G. 
Lindzey. Reading, Mass.: addison-Wesiey, 1968. 

Mullis, Ina V.S. "Using the Primary Trait System for Evaluating Writing," no. 
lO-W-51. Denver, Colo.: National Assessment of Educational Progress, 
Education Commission of the States, 1980. ISBN 0-89398-396-9. 

Mullis, Ina V.S. and John C. Mellon. "Guidelines for Three WSys of Evaluatino 
Writing: Syntax, Cohesion and Mechanics," no. lO-W-50. Denver, Cbio.: 
National Assessment of Educational Progress, Education Cbmmission of the 
States, 1980. ISBN 0-89398-395-0. 

Rogers, W.T. et al. "Assessment of Nonresponse Bias in Sample Surveys: An 
Exampla From National Assessment,^ Journal of Educational .Measurement, vol. 
14, no. 4, 1977. " • 

The Th ird Assessment of Writing: 1978-79 Released Exercise Set , no. lO-W-25, 
19.3-79 Assessment. Denver, Colo.: National Assessment of Educational 
Progress, Education Commission of the States, 1981. ISBN 0-89398-394-2. 

■^key,^ John W. Exploratory Cata Analysis . Reading, Mass.: .Add i son-Wesley, 



Writing Achievenient, 1969-79: Results From the Third National Writing 
Assessment, Volume I — 17-Year-Olds , Report no. 10-W-Ol, 1969-70, 1973-74 
and 1978-79 Assessments. Denver, Colo.: National Assessment: of Educational 
Progress, Education Commission of the States, 1980. ISBN 0-8935'^ -4 00-0. 

Writing Achievement, 1969-79: Results From the Third National Writing 
Assessment, Volume II — 13-Year-Olds , Repcrt no. lO-lV-02, 1969-70, 1973-"4 
and 1978-79 Assessments. Denver, Colo.: National Assessment of Educational 
Progres:-., Education Commission of the States, 1980. ISBN 0-89398-401-9. 

Writing Achievement, 1969-79; Results From the Third National Writing 
Assessment, Volume III — 9-Year-Olds , Report no. lO-W-03, 1969-70, 1973-74 
and 19"8-79 Assessments. Denver, Colo.: National Assessment of Educational 
Progress, Education Commission of the States, 1980. ISBN 0-89398-402-7. 

writing .^techanics, 1969-74 , Report nc. 05-^V-Ol, 1969-70 and 1973-74 
Assessments. Denver, Colo.: Nat )nal Assessment of Educational Progress, 
Education Commission of the States, 1975. ERIC no. 113 736. ISBN 

0-89398-383-7. 

writing Objsctives, Second Assessment , 1973-74 .Assessment. Denver, Cblo.r 
National Assessment of Educational Progress, Education Commission of the 
States, 1972. ERIC no. ED 072 460. ISBN 0-89398-381-0. 



88 



(Continued From Inside Front Cover) 

In addition to the above reports. National Assessnnent has produced reports in the areas of social studies, citizenship, 
music, art mathematics, science and career and occupation?! development. A complete publications list and ordering 
information are available from the address beiow. 

NATIONAL ASSESSMEMT OF EDUCATIONAL PROGRESS 
Suite 700, 1860 Lincoln Street 
Denver, Colorado 80295 



